[==========] Running 38 tests from 2 test suites.
[----------] Global test environment set-up.
[----------] 37 tests from ExternalMiniClusterTest
[ RUN ] ExternalMiniClusterTest.TestKerberosReacquire
Loading random data
Initializing database '/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:50:03 dist-test-slave-vz23 krb5kdc[292](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:50:03 dist-test-slave-vz23 krb5kdc[292](info): set up 2 sockets
Sep 29 01:50:03 dist-test-slave-vz23 krb5kdc[292](info): commencing operation
krb5kdc: starting...
WARNING: Logging before InitGoogleLogging() is written to STDERR
W20260929 01:50:05.574388 285 mini_kdc.cc:121] Time spent starting KDC: real 2.127s user 0.002s sys 0.015s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 01:50:05 dist-test-slave-vz23 krb5kdc[292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:50:05Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:50:05Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:50:05.840281 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:36401
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:43933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:36401 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:50:06.187072 308 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:06.187451 308 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:06.187606 308 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:06.196583 308 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:50:06.196723 308 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:06.196818 308 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:50:06.196908 308 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:50:06.207782 308 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:43933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:36401
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:36401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:06.209415 308 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:06.210808 308 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:06.219951 313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:06.220116 314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:06.220654 316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:06.221067 308 server_base.cc:1061] running on GCE node
I20260929 01:50:06.221783 308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:06.223093 308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:06.224306 308 hybrid_clock.cc:648] HybridClock initialized: now 1790646606224228 us; error 76 us; skew 500 ppm
I20260929 01:50:06.224664 308 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:50:06 dist-test-slave-vz23 krb5kdc[292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:50:06.227722 308 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:50:06.229359 308 webserver.cc:533] Webserver started at http://127.0.71.126:35079/ using document root <none> and password file <none>
I20260929 01:50:06.230058 308 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:06.230186 308 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:06.230499 308 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:50:06.233330 308 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "cebaed45400549e68ca0ca9df2b0794f"
format_stamp: "Formatted at 2026-09-29 01:50:06 on dist-test-slave-vz23"
I20260929 01:50:06.234078 308 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "cebaed45400549e68ca0ca9df2b0794f"
format_stamp: "Formatted at 2026-09-29 01:50:06 on dist-test-slave-vz23"
I20260929 01:50:06.239457 308 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 01:50:06.242740 323 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:06.244091 308 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 01:50:06.244253 308 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "cebaed45400549e68ca0ca9df2b0794f"
format_stamp: "Formatted at 2026-09-29 01:50:06 on dist-test-slave-vz23"
I20260929 01:50:06.244436 308 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:06.259414 308 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:06.263286 308 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:06.263577 308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:06.285717 308 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:36401
I20260929 01:50:06.285738 374 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:36401 every 8 connection(s)
I20260929 01:50:06.287779 308 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:50:06.292040 375 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:50:06.298327 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 308
I20260929 01:50:06.298648 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:50:06.301124 375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f: Bootstrap starting.
I20260929 01:50:06.305171 375 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f: Neither blocks nor log segments found. Creating new log.
I20260929 01:50:06.306474 375 log.cc:826] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f: Log is configured to *not* fsync() on all Append() calls
I20260929 01:50:06.309504 375 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f: No bootstrap required, opened a new log
I20260929 01:50:06.317282 375 raft_consensus.cc:359] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebaed45400549e68ca0ca9df2b0794f" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36401 } }
I20260929 01:50:06.317631 375 raft_consensus.cc:385] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:50:06.317762 375 raft_consensus.cc:740] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: cebaed45400549e68ca0ca9df2b0794f, State: Initialized, Role: FOLLOWER
I20260929 01:50:06.318531 375 consensus_queue.cc:260] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebaed45400549e68ca0ca9df2b0794f" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36401 } }
I20260929 01:50:06.318745 375 raft_consensus.cc:399] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:50:06.318897 375 raft_consensus.cc:493] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:50:06.319061 375 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:50:06.321348 375 raft_consensus.cc:515] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebaed45400549e68ca0ca9df2b0794f" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36401 } }
I20260929 01:50:06.321859 375 leader_election.cc:304] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cebaed45400549e68ca0ca9df2b0794f; no voters:
I20260929 01:50:06.322575 375 leader_election.cc:290] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 01:50:06.322679 380 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:50:06.324256 380 raft_consensus.cc:697] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [term 1 LEADER]: Becoming Leader. State: Replica: cebaed45400549e68ca0ca9df2b0794f, State: Running, Role: LEADER
I20260929 01:50:06.324949 380 consensus_queue.cc:237] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebaed45400549e68ca0ca9df2b0794f" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36401 } }
I20260929 01:50:06.325654 375 sys_catalog.cc:565] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:50:06.327854 381 sys_catalog.cc:455] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "cebaed45400549e68ca0ca9df2b0794f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebaed45400549e68ca0ca9df2b0794f" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36401 } } }
I20260929 01:50:06.328157 381 sys_catalog.cc:458] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [sys.catalog]: This master's current role is: LEADER
Sep 29 01:50:06 dist-test-slave-vz23 krb5kdc[292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646605, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:50:06.335655 382 sys_catalog.cc:455] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cebaed45400549e68ca0ca9df2b0794f. Latest consensus state: current_term: 1 leader_uuid: "cebaed45400549e68ca0ca9df2b0794f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebaed45400549e68ca0ca9df2b0794f" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36401 } } }
I20260929 01:50:06.335997 382 sys_catalog.cc:458] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:06.337098 385 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:50:06.343349 385 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:50:06.354878 385 catalog_manager.cc:1396] Generated new cluster ID: d0062c41de894be1b45c874ec7549d56
I20260929 01:50:06.355007 385 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:50:06.375950 385 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 01:50:06.378100 385 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:50:06.407554 385 catalog_manager.cc:6219] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f: Generated new TSK 0
I20260929 01:50:06.408627 385 catalog_manager.cc:1563] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:50:06.500339 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:36401
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:43933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:50:06.854811 404 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:06.855189 404 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:06.855374 404 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:06.865228 404 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:06.865455 404 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:50:06.877698 404 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:43933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:36401
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:06.879261 404 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:06.880606 404 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:06.891220 410 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:06.891317 412 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:06.891258 409 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:06.891906 404 server_base.cc:1061] running on GCE node
I20260929 01:50:06.892724 404 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:06.893594 404 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:06.894820 404 hybrid_clock.cc:648] HybridClock initialized: now 1790646606894777 us; error 35 us; skew 500 ppm
I20260929 01:50:06.895153 404 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:50:06 dist-test-slave-vz23 krb5kdc[292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:50:06.898988 404 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:50:06.900743 404 webserver.cc:533] Webserver started at http://127.0.71.65:37601/ using document root <none> and password file <none>
I20260929 01:50:06.901454 404 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:06.901551 404 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:06.902110 404 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:50:06.904742 404 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "0ccce6b021154b6c8b1ea0b64facb9c4"
format_stamp: "Formatted at 2026-09-29 01:50:06 on dist-test-slave-vz23"
I20260929 01:50:06.905601 404 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "0ccce6b021154b6c8b1ea0b64facb9c4"
format_stamp: "Formatted at 2026-09-29 01:50:06 on dist-test-slave-vz23"
I20260929 01:50:06.911420 404 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:50:06.914846 419 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:06.916185 404 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 01:50:06.916364 404 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "0ccce6b021154b6c8b1ea0b64facb9c4"
format_stamp: "Formatted at 2026-09-29 01:50:06 on dist-test-slave-vz23"
I20260929 01:50:06.916615 404 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:06.936753 404 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:06.940836 404 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:06.941139 404 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:06.942101 404 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:06.943714 404 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:06.943790 404 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:06.943975 404 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:06.944042 404 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:06.983039 404 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:45875
I20260929 01:50:06.983080 531 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:45875 every 8 connection(s)
I20260929 01:50:06.984411 404 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:50:06.989614 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 404
I20260929 01:50:06.989827 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
Sep 29 01:50:06 dist-test-slave-vz23 krb5kdc[292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646606, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:50:07.007766 532 heartbeater.cc:344] Connected to a master server at 127.0.71.126:36401
I20260929 01:50:07.008240 532 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:07.009183 532 heartbeater.cc:507] Master 127.0.71.126:36401 requested a full tablet report, sending...
I20260929 01:50:07.011336 339 ts_manager.cc:194] Registered new tserver with Master: 0ccce6b021154b6c8b1ea0b64facb9c4 (127.0.71.65:45875)
I20260929 01:50:07.012948 339 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:41401
I20260929 01:50:07.020793 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:50:08.016260 532 heartbeater.cc:499] Master 127.0.71.126:36401 was elected leader, sending a full tablet report...
Sep 29 01:50:21 dist-test-slave-vz23 krb5kdc[292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:50:21.229840 319 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 01:50:21 dist-test-slave-vz23 krb5kdc[292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646621, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:50:21.901221 415 init.cc:295] Successfully reacquired a new kerberos TGT
Sep 29 01:50:23 dist-test-slave-vz23 krb5kdc[292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
I20260929 01:50:23.051937 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 308
I20260929 01:50:23.053607 370 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:23.160022 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 308
I20260929 01:50:23.181461 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:36401
--webserver_interface=127.0.71.126
--webserver_port=35079
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:43933
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:36401 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:50:23.503127 539 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:23.503439 539 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:23.503523 539 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:23.511905 539 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:50:23.512015 539 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:23.512080 539 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:50:23.512130 539 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:50:23.522822 539 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:43933
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:36401
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:36401
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=35079
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:23.524629 539 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:23.525907 539 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:23.534510 544 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:23.534551 545 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:23.535275 547 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:23.535702 539 server_base.cc:1061] running on GCE node
I20260929 01:50:23.536379 539 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:23.537848 539 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:23.539078 539 hybrid_clock.cc:648] HybridClock initialized: now 1790646623539028 us; error 38 us; skew 500 ppm
I20260929 01:50:23.539546 539 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:50:23 dist-test-slave-vz23 krb5kdc[292](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646623, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:50:23.543099 539 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:50:23.544965 539 webserver.cc:533] Webserver started at http://127.0.71.126:35079/ using document root <none> and password file <none>
I20260929 01:50:23.545686 539 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:23.545795 539 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:23.551687 539 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 01:50:23.554800 554 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:23.556201 539 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:50:23.556373 539 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "cebaed45400549e68ca0ca9df2b0794f"
format_stamp: "Formatted at 2026-09-29 01:50:06 on dist-test-slave-vz23"
I20260929 01:50:23.557292 539 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:23.580011 539 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:23.591926 539 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:23.592284 539 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:23.612856 539 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:36401
I20260929 01:50:23.612875 605 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:36401 every 8 connection(s)
I20260929 01:50:23.615180 539 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestKerberosReacquire.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:50:23.618814 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 539
I20260929 01:50:23.623347 606 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:50:23.626518 606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f: Bootstrap starting.
Sep 29 01:50:23 dist-test-slave-vz23 krb5kdc[292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:50:23.644647 606 log.cc:826] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f: Log is configured to *not* fsync() on all Append() calls
I20260929 01:50:23.652243 606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:50:23.652769 606 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f: Bootstrap complete.
I20260929 01:50:23.658928 606 raft_consensus.cc:359] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebaed45400549e68ca0ca9df2b0794f" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36401 } }
I20260929 01:50:23.659567 606 raft_consensus.cc:740] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: cebaed45400549e68ca0ca9df2b0794f, State: Initialized, Role: FOLLOWER
I20260929 01:50:23.660244 606 consensus_queue.cc:260] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebaed45400549e68ca0ca9df2b0794f" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36401 } }
I20260929 01:50:23.660475 606 raft_consensus.cc:399] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [term 1 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 01:50:23.660616 606 raft_consensus.cc:493] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [term 1 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 01:50:23.660813 606 raft_consensus.cc:3060] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:50:23.664354 606 raft_consensus.cc:515] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebaed45400549e68ca0ca9df2b0794f" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36401 } }
I20260929 01:50:23.664913 606 leader_election.cc:304] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: cebaed45400549e68ca0ca9df2b0794f; no voters:
I20260929 01:50:23.665591 606 leader_election.cc:290] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [CANDIDATE]: Term 2 election: Requested vote from peers
I20260929 01:50:23.665727 611 raft_consensus.cc:2804] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:50:23.667248 611 raft_consensus.cc:697] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [term 2 LEADER]: Becoming Leader. State: Replica: cebaed45400549e68ca0ca9df2b0794f, State: Running, Role: LEADER
I20260929 01:50:23.667982 606 sys_catalog.cc:565] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:50:23.667874 611 consensus_queue.cc:237] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebaed45400549e68ca0ca9df2b0794f" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36401 } }
I20260929 01:50:23.670758 613 sys_catalog.cc:455] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [sys.catalog]: SysCatalogTable state changed. Reason: New leader cebaed45400549e68ca0ca9df2b0794f. Latest consensus state: current_term: 2 leader_uuid: "cebaed45400549e68ca0ca9df2b0794f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebaed45400549e68ca0ca9df2b0794f" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36401 } } }
I20260929 01:50:23.671080 613 sys_catalog.cc:458] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:23.671365 612 sys_catalog.cc:455] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 leader_uuid: "cebaed45400549e68ca0ca9df2b0794f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "cebaed45400549e68ca0ca9df2b0794f" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36401 } } }
I20260929 01:50:23.671676 612 sys_catalog.cc:458] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:23.673579 618 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:50:23.680513 618 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:50:23.682164 618 catalog_manager.cc:1308] Loaded cluster ID: d0062c41de894be1b45c874ec7549d56
I20260929 01:50:23.682245 618 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:50:23.684991 618 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:50:23.686088 618 catalog_manager.cc:6230] T 00000000000000000000000000000000 P cebaed45400549e68ca0ca9df2b0794f: Loaded TSK: 0
I20260929 01:50:23.687567 618 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:50:24.058188 571 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "0ccce6b021154b6c8b1ea0b64facb9c4" instance_seqno: 1790646606973979) as {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:58391; Asking this server to re-register.
I20260929 01:50:24.058966 532 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:24.059234 532 heartbeater.cc:507] Master 127.0.71.126:36401 requested a full tablet report, sending...
I20260929 01:50:24.060585 571 ts_manager.cc:194] Registered new tserver with Master: 0ccce6b021154b6c8b1ea0b64facb9c4 (127.0.71.65:45875)
I20260929 01:50:24.068815 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:50:24.069068 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 404
Sep 29 01:50:24 dist-test-slave-vz23 krb5kdc[292](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646623, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 01:50:24.086596 527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:24.176599 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 404
I20260929 01:50:24.200227 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 539
I20260929 01:50:24.201479 601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:24.303977 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 539
2026-09-29T01:50:24Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestKerberosReacquire (20914 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/0
I20260929 01:50:24.347680 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:42419
--webserver_interface=127.0.71.126
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033 with env {}
W20260929 01:50:24.674764 634 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:24.675160 634 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:24.675297 634 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:24.683825 634 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:50:24.683956 634 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:24.684070 634 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:50:24.684156 634 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:50:24.694303 634 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:42419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:24.695936 634 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:24.697727 634 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:24.705706 639 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:24.705633 640 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:24.706452 642 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:24.707011 634 server_base.cc:1061] running on GCE node
I20260929 01:50:24.707703 634 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:50:24.707837 634 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:50:24.707899 634 hybrid_clock.cc:648] HybridClock initialized: now 1790646624707883 us; error 0 us; skew 500 ppm
I20260929 01:50:24.708182 634 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:24.710991 634 webserver.cc:533] Webserver started at http://127.0.71.126:46793/ using document root <none> and password file <none>
I20260929 01:50:24.711664 634 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:24.711746 634 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:24.712016 634 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:50:24.714277 634 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "0ea38b4fa64841e8a36df8bb12a20110"
format_stamp: "Formatted at 2026-09-29 01:50:24 on dist-test-slave-vz23"
I20260929 01:50:24.714942 634 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "0ea38b4fa64841e8a36df8bb12a20110"
format_stamp: "Formatted at 2026-09-29 01:50:24 on dist-test-slave-vz23"
I20260929 01:50:24.719928 634 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 01:50:24.723273 647 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:24.724856 634 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:50:24.725044 634 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "0ea38b4fa64841e8a36df8bb12a20110"
format_stamp: "Formatted at 2026-09-29 01:50:24 on dist-test-slave-vz23"
I20260929 01:50:24.725238 634 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:24.751072 634 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:24.752089 634 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:24.752415 634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:24.774106 634 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:42419
I20260929 01:50:24.774158 698 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:42419 every 8 connection(s)
I20260929 01:50:24.775774 634 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:50:24.780249 699 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:50:24.781059 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 634
I20260929 01:50:24.781347 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:50:24.782569 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:43325
--webserver_interface=127.0.71.125
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033 with env {}
I20260929 01:50:24.786775 699 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:50:24.805226 699 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:50:24.806362 648 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.125:43325: connect: Connection refused (error 111)
W20260929 01:50:24.807972 699 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:43325: Network error: Client connection negotiation failed: client connection to 127.0.71.125:43325: connect: Connection refused (error 111)
I20260929 01:50:24.857414 699 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } attempt: 1
W20260929 01:50:24.859707 699 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:43325: Network error: Client connection negotiation failed: client connection to 127.0.71.125:43325: connect: Connection refused (error 111)
I20260929 01:50:24.928040 699 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } attempt: 2
W20260929 01:50:24.930302 699 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:43325: Network error: Client connection negotiation failed: client connection to 127.0.71.125:43325: connect: Connection refused (error 111)
I20260929 01:50:25.021603 699 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } attempt: 3
W20260929 01:50:25.023610 699 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:43325: Network error: Client connection negotiation failed: client connection to 127.0.71.125:43325: connect: Connection refused (error 111)
W20260929 01:50:25.123236 701 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:25.123587 701 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:25.123672 701 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:25.132148 701 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:50:25.132257 701 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:25.132321 701 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:50:25.132367 701 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:50:25.142506 701 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:43325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:25.144091 701 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:25.145856 701 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:25.155121 712 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:25.155221 710 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:25.155232 709 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:25.156039 701 server_base.cc:1061] running on GCE node
I20260929 01:50:25.156838 701 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:50:25.157039 701 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:50:25.157112 701 hybrid_clock.cc:648] HybridClock initialized: now 1790646625157095 us; error 0 us; skew 500 ppm
I20260929 01:50:25.157438 701 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:25.160081 701 webserver.cc:533] Webserver started at http://127.0.71.125:44163/ using document root <none> and password file <none>
I20260929 01:50:25.160887 701 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:25.160976 701 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:25.161356 701 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:50:25.163650 701 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/data/instance:
uuid: "a067d22fc04447ed8804d60c167c0bfc"
format_stamp: "Formatted at 2026-09-29 01:50:25 on dist-test-slave-vz23"
I20260929 01:50:25.164414 701 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/wal/instance:
uuid: "a067d22fc04447ed8804d60c167c0bfc"
format_stamp: "Formatted at 2026-09-29 01:50:25 on dist-test-slave-vz23"
I20260929 01:50:25.166873 699 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } attempt: 4
W20260929 01:50:25.168927 699 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:43325: Network error: Client connection negotiation failed: client connection to 127.0.71.125:43325: connect: Connection refused (error 111)
I20260929 01:50:25.170125 701 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 01:50:25.173929 717 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:25.175279 701 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 01:50:25.175477 701 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "a067d22fc04447ed8804d60c167c0bfc"
format_stamp: "Formatted at 2026-09-29 01:50:25 on dist-test-slave-vz23"
I20260929 01:50:25.175716 701 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:25.201277 701 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:25.202189 701 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:25.202479 701 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:25.221562 701 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:43325
I20260929 01:50:25.221588 768 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:43325 every 8 connection(s)
I20260929 01:50:25.223206 701 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:50:25.227186 769 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:50:25.230984 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 701
I20260929 01:50:25.231227 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:50:25.232192 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:40033
--webserver_interface=127.0.71.124
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033 with env {}
I20260929 01:50:25.233660 769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:50:25.246665 769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:50:25.255899 769 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:50:25.257354 720 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40033: connect: Connection refused (error 111)
W20260929 01:50:25.258914 769 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:40033: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40033: connect: Connection refused (error 111)
I20260929 01:50:25.308322 769 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } attempt: 1
W20260929 01:50:25.310391 769 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:40033: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40033: connect: Connection refused (error 111)
I20260929 01:50:25.378764 769 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } attempt: 2
W20260929 01:50:25.381282 769 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:40033: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40033: connect: Connection refused (error 111)
I20260929 01:50:25.468216 699 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } attempt: 5
I20260929 01:50:25.472513 769 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } attempt: 3
W20260929 01:50:25.474601 769 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:40033: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40033: connect: Connection refused (error 111)
I20260929 01:50:25.475577 699 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:50:25.477759 699 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:40033: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40033: connect: Connection refused (error 111)
I20260929 01:50:25.528988 699 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } attempt: 1
W20260929 01:50:25.531311 699 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:40033: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40033: connect: Connection refused (error 111)
W20260929 01:50:25.568897 771 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:25.569207 771 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:25.569276 771 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:25.578086 771 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:50:25.578190 771 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:25.578289 771 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:50:25.578373 771 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:50:25.588313 771 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:40033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:25.589967 771 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:25.592592 771 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260929 01:50:25.599589 699 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } attempt: 2
W20260929 01:50:25.600734 779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:25.600804 780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:25.602242 699 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:40033: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40033: connect: Connection refused (error 111)
W20260929 01:50:25.602440 782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:25.602918 771 server_base.cc:1061] running on GCE node
I20260929 01:50:25.603703 771 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:50:25.603845 771 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:50:25.603909 771 hybrid_clock.cc:648] HybridClock initialized: now 1790646625603892 us; error 0 us; skew 500 ppm
I20260929 01:50:25.604190 771 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:25.606844 771 webserver.cc:533] Webserver started at http://127.0.71.124:36611/ using document root <none> and password file <none>
I20260929 01:50:25.607491 771 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:25.607574 771 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:25.608270 771 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:50:25.610630 771 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/data/instance:
uuid: "96a59e909fb44ae1bc1b7ac327df09b5"
format_stamp: "Formatted at 2026-09-29 01:50:25 on dist-test-slave-vz23"
I20260929 01:50:25.611320 771 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/wal/instance:
uuid: "96a59e909fb44ae1bc1b7ac327df09b5"
format_stamp: "Formatted at 2026-09-29 01:50:25 on dist-test-slave-vz23"
I20260929 01:50:25.616431 771 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 01:50:25.617818 769 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } attempt: 4
W20260929 01:50:25.620072 769 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:40033: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40033: connect: Connection refused (error 111)
I20260929 01:50:25.620205 787 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:25.622017 771 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:50:25.622213 771 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "96a59e909fb44ae1bc1b7ac327df09b5"
format_stamp: "Formatted at 2026-09-29 01:50:25 on dist-test-slave-vz23"
I20260929 01:50:25.622467 771 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:25.665176 771 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:25.666143 771 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:25.666445 771 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:25.688627 771 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:40033
I20260929 01:50:25.688603 838 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:40033 every 8 connection(s)
I20260929 01:50:25.690208 771 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:50:25.690680 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 771
I20260929 01:50:25.690882 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:50:25.696033 839 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:50:25.703840 839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:50:25.708485 699 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } attempt: 3
I20260929 01:50:25.716189 839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:50:25.721184 699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: Bootstrap starting.
I20260929 01:50:25.723310 839 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:50:25.725265 699 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: Neither blocks nor log segments found. Creating new log.
I20260929 01:50:25.726361 699 log.cc:826] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: Log is configured to *not* fsync() on all Append() calls
I20260929 01:50:25.729523 699 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: No bootstrap required, opened a new log
I20260929 01:50:25.733032 839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5: Bootstrap starting.
I20260929 01:50:25.735622 699 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } }
I20260929 01:50:25.735852 699 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:50:25.735987 699 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea38b4fa64841e8a36df8bb12a20110, State: Initialized, Role: FOLLOWER
I20260929 01:50:25.736579 839 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5: Neither blocks nor log segments found. Creating new log.
I20260929 01:50:25.736815 699 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } }
I20260929 01:50:25.737788 839 log.cc:826] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5: Log is configured to *not* fsync() on all Append() calls
I20260929 01:50:25.737744 846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:25.738049 846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:25.738689 699 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:50:25.740628 839 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5: No bootstrap required, opened a new log
I20260929 01:50:25.746083 839 raft_consensus.cc:359] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } }
I20260929 01:50:25.746341 839 raft_consensus.cc:385] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:50:25.746484 839 raft_consensus.cc:740] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96a59e909fb44ae1bc1b7ac327df09b5, State: Initialized, Role: FOLLOWER
I20260929 01:50:25.747144 839 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } }
I20260929 01:50:25.748154 856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:25.748970 856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:25.749508 839 sys_catalog.cc:565] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:50:25.755051 870 catalog_manager.cc:818] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260929 01:50:25.755331 653 tablet.cc:2433] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:50:25.759742 869 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:50:25.759891 869 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:50:25.760900 634 master_runner.cc:187] Error getting master registration for 127.0.71.125:43325: OK, instance_id { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" instance_seqno: 1790646625216396 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:50:25.767187 634 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:50:25.762454 771 master_runner.cc:187] Error getting master registration for 127.0.71.125:43325: OK, instance_id { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" instance_seqno: 1790646625216396 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:50:25.768832 771 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:50:25.918681 793 tablet.cc:2433] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:50:25.919351 769 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } attempt: 5
I20260929 01:50:25.931979 769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc: Bootstrap starting.
I20260929 01:50:25.936049 769 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc: Neither blocks nor log segments found. Creating new log.
I20260929 01:50:25.937853 769 log.cc:826] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc: Log is configured to *not* fsync() on all Append() calls
I20260929 01:50:25.941087 769 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc: No bootstrap required, opened a new log
I20260929 01:50:25.946128 769 raft_consensus.cc:359] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } }
I20260929 01:50:25.946411 769 raft_consensus.cc:385] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:50:25.946555 769 raft_consensus.cc:740] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: a067d22fc04447ed8804d60c167c0bfc, State: Initialized, Role: FOLLOWER
I20260929 01:50:25.947283 769 consensus_queue.cc:260] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } }
I20260929 01:50:25.948335 873 sys_catalog.cc:455] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:25.948643 873 sys_catalog.cc:458] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:25.949343 873 raft_consensus.cc:493] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:50:25.949854 769 sys_catalog.cc:565] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:50:25.949968 873 raft_consensus.cc:515] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } }
I20260929 01:50:25.952641 674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a067d22fc04447ed8804d60c167c0bfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea38b4fa64841e8a36df8bb12a20110" is_pre_election: true
I20260929 01:50:25.953109 674 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a067d22fc04447ed8804d60c167c0bfc in term 0.
I20260929 01:50:25.953920 718 leader_election.cc:304] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea38b4fa64841e8a36df8bb12a20110, a067d22fc04447ed8804d60c167c0bfc; no voters:
I20260929 01:50:25.954077 814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a067d22fc04447ed8804d60c167c0bfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" is_pre_election: true
I20260929 01:50:25.954617 814 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate a067d22fc04447ed8804d60c167c0bfc in term 0.
W20260929 01:50:25.955472 723 tablet.cc:2433] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:50:25.956629 873 leader_election.cc:290] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0ea38b4fa64841e8a36df8bb12a20110 (127.0.71.126:42419), 96a59e909fb44ae1bc1b7ac327df09b5 (127.0.71.124:40033)
I20260929 01:50:25.956845 873 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:50:25.956969 873 raft_consensus.cc:493] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:50:25.957067 873 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:50:25.960345 873 raft_consensus.cc:515] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } }
I20260929 01:50:25.961530 814 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a067d22fc04447ed8804d60c167c0bfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "96a59e909fb44ae1bc1b7ac327df09b5"
I20260929 01:50:25.961724 814 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:50:25.965806 814 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a067d22fc04447ed8804d60c167c0bfc in term 1.
I20260929 01:50:25.968086 720 leader_election.cc:304] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 96a59e909fb44ae1bc1b7ac327df09b5, a067d22fc04447ed8804d60c167c0bfc; no voters:
I20260929 01:50:25.968497 873 leader_election.cc:290] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [CANDIDATE]: Term 1 election: Requested vote from peers 0ea38b4fa64841e8a36df8bb12a20110 (127.0.71.126:42419), 96a59e909fb44ae1bc1b7ac327df09b5 (127.0.71.124:40033)
I20260929 01:50:25.968729 873 raft_consensus.cc:2804] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:50:25.968865 873 raft_consensus.cc:697] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [term 1 LEADER]: Becoming Leader. State: Replica: a067d22fc04447ed8804d60c167c0bfc, State: Running, Role: LEADER
I20260929 01:50:25.969389 873 consensus_queue.cc:237] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } }
I20260929 01:50:25.971657 674 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a067d22fc04447ed8804d60c167c0bfc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0ea38b4fa64841e8a36df8bb12a20110"
I20260929 01:50:25.971930 674 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:50:25.976850 674 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate a067d22fc04447ed8804d60c167c0bfc in term 1.
I20260929 01:50:25.979236 876 sys_catalog.cc:455] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader a067d22fc04447ed8804d60c167c0bfc. Latest consensus state: current_term: 1 leader_uuid: "a067d22fc04447ed8804d60c167c0bfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:25.979619 876 sys_catalog.cc:458] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [sys.catalog]: This master's current role is: LEADER
W20260929 01:50:25.995987 887 catalog_manager.cc:1607] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:50:25.996241 887 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:50:25.996420 880 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:50:25.998925 880 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:50:26.006258 814 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [term 1 FOLLOWER]: Refusing update from remote peer a067d22fc04447ed8804d60c167c0bfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:50:26.006253 674 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [term 1 FOLLOWER]: Refusing update from remote peer a067d22fc04447ed8804d60c167c0bfc: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:50:26.007302 873 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:50:26.007979 873 consensus_queue.cc:1048] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [LEADER]: Connected to new peer: Peer: permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:50:26.033850 846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a067d22fc04447ed8804d60c167c0bfc. Latest consensus state: current_term: 1 leader_uuid: "a067d22fc04447ed8804d60c167c0bfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:26.034085 846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:26.034917 856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a067d22fc04447ed8804d60c167c0bfc. Latest consensus state: current_term: 1 leader_uuid: "a067d22fc04447ed8804d60c167c0bfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:26.035181 856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:26.039119 873 sys_catalog.cc:455] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a067d22fc04447ed8804d60c167c0bfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:26.039391 873 sys_catalog.cc:458] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:26.039628 873 sys_catalog.cc:455] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a067d22fc04447ed8804d60c167c0bfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:26.039858 873 sys_catalog.cc:458] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:26.040375 880 catalog_manager.cc:1396] Generated new cluster ID: 8f5451ce215a49c0b44c8b9a8e05085a
I20260929 01:50:26.040498 880 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:50:26.042515 891 mvcc.cc:204] Tried to move back new op lower bound from 7334488580111458304 to 7334488579977854976. Current Snapshot: MvccSnapshot[applied={T|T < 7334488580111458304}]
I20260929 01:50:26.048915 856 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a067d22fc04447ed8804d60c167c0bfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:26.049139 856 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:26.057179 846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "a067d22fc04447ed8804d60c167c0bfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:26.057413 846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:26.063730 880 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 01:50:26.065182 880 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:50:26.077510 880 catalog_manager.cc:6219] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc: Generated new TSK 0
I20260929 01:50:26.078603 880 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:50:26.093904 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--time_source=system_unsync with env {}
W20260929 01:50:26.503953 897 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:26.504258 897 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:26.504328 897 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:26.512614 897 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:26.512780 897 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:50:26.523032 897 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:26.524662 897 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:26.526317 897 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:26.536070 902 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:26.536317 903 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:26.536263 905 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:26.537213 897 server_base.cc:1061] running on GCE node
I20260929 01:50:26.537779 897 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:50:26.537921 897 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:50:26.537983 897 hybrid_clock.cc:648] HybridClock initialized: now 1790646626537967 us; error 0 us; skew 500 ppm
I20260929 01:50:26.538254 897 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:26.540561 897 webserver.cc:533] Webserver started at http://127.0.71.65:38429/ using document root <none> and password file <none>
I20260929 01:50:26.541252 897 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:26.541334 897 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:26.541635 897 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:50:26.543969 897 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "9f86d7b4e82e43079d9a7d728aca3860"
format_stamp: "Formatted at 2026-09-29 01:50:26 on dist-test-slave-vz23"
I20260929 01:50:26.544790 897 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "9f86d7b4e82e43079d9a7d728aca3860"
format_stamp: "Formatted at 2026-09-29 01:50:26 on dist-test-slave-vz23"
I20260929 01:50:26.549985 897 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 01:50:26.553321 910 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:26.555042 897 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:50:26.555182 897 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "9f86d7b4e82e43079d9a7d728aca3860"
format_stamp: "Formatted at 2026-09-29 01:50:26 on dist-test-slave-vz23"
I20260929 01:50:26.555343 897 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:26.575191 897 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:26.576109 897 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:26.576373 897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:26.577342 897 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:26.578948 897 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:26.579031 897 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:26.579159 897 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:26.579242 897 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:26.617882 897 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:39199
I20260929 01:50:26.617925 1023 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:39199 every 8 connection(s)
I20260929 01:50:26.619719 897 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:50:26.628947 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 897
I20260929 01:50:26.629202 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
I20260929 01:50:26.638553 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.66:0
--local_ip_for_outbound_sockets=127.0.71.66
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--time_source=system_unsync with env {}
I20260929 01:50:26.656647 1024 heartbeater.cc:344] Connected to a master server at 127.0.71.124:40033
I20260929 01:50:26.656687 1025 heartbeater.cc:344] Connected to a master server at 127.0.71.126:42419
I20260929 01:50:26.656658 1026 heartbeater.cc:344] Connected to a master server at 127.0.71.125:43325
I20260929 01:50:26.657137 1024 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:26.657132 1026 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:26.657132 1025 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:26.658531 1026 heartbeater.cc:507] Master 127.0.71.125:43325 requested a full tablet report, sending...
I20260929 01:50:26.658552 1024 heartbeater.cc:507] Master 127.0.71.124:40033 requested a full tablet report, sending...
I20260929 01:50:26.658586 1025 heartbeater.cc:507] Master 127.0.71.126:42419 requested a full tablet report, sending...
I20260929 01:50:26.661263 804 ts_manager.cc:194] Registered new tserver with Master: 9f86d7b4e82e43079d9a7d728aca3860 (127.0.71.65:39199)
I20260929 01:50:26.661265 734 ts_manager.cc:194] Registered new tserver with Master: 9f86d7b4e82e43079d9a7d728aca3860 (127.0.71.65:39199)
I20260929 01:50:26.661338 663 ts_manager.cc:194] Registered new tserver with Master: 9f86d7b4e82e43079d9a7d728aca3860 (127.0.71.65:39199)
I20260929 01:50:26.663179 734 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:33277
I20260929 01:50:26.760811 870 catalog_manager.cc:1308] Loaded cluster ID: 8f5451ce215a49c0b44c8b9a8e05085a
I20260929 01:50:26.760947 870 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: loading cluster ID for follower catalog manager: success
I20260929 01:50:26.762380 869 catalog_manager.cc:1308] Loaded cluster ID: 8f5451ce215a49c0b44c8b9a8e05085a
I20260929 01:50:26.762476 869 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5: loading cluster ID for follower catalog manager: success
I20260929 01:50:26.764766 870 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: acquiring CA information for follower catalog manager: success
I20260929 01:50:26.765985 869 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5: acquiring CA information for follower catalog manager: success
I20260929 01:50:26.767133 870 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:50:26.769016 869 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 01:50:26.997210 1032 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:26.997562 1032 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:26.997696 1032 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:27.010249 1032 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:27.010517 1032 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.66
I20260929 01:50:27.021739 1032 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:27.023424 1032 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:27.025310 1032 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:27.035950 1041 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:27.037287 1040 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:27.037577 1043 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:27.039126 1032 server_base.cc:1061] running on GCE node
I20260929 01:50:27.039835 1032 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:50:27.040035 1032 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:50:27.040125 1032 hybrid_clock.cc:648] HybridClock initialized: now 1790646627040107 us; error 0 us; skew 500 ppm
I20260929 01:50:27.040493 1032 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:27.042954 1032 webserver.cc:533] Webserver started at http://127.0.71.66:37575/ using document root <none> and password file <none>
I20260929 01:50:27.043751 1032 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:27.043838 1032 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:27.044246 1032 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:50:27.046862 1032 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/data/instance:
uuid: "a8b28722d60646f7adfa970805c79cf5"
format_stamp: "Formatted at 2026-09-29 01:50:27 on dist-test-slave-vz23"
I20260929 01:50:27.047638 1032 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/wal/instance:
uuid: "a8b28722d60646f7adfa970805c79cf5"
format_stamp: "Formatted at 2026-09-29 01:50:27 on dist-test-slave-vz23"
I20260929 01:50:27.053923 1032 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 01:50:27.057617 1048 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:27.059165 1032 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 01:50:27.059365 1032 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/wal
uuid: "a8b28722d60646f7adfa970805c79cf5"
format_stamp: "Formatted at 2026-09-29 01:50:27 on dist-test-slave-vz23"
I20260929 01:50:27.059589 1032 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:27.083289 1032 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:27.084223 1032 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:27.084568 1032 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:27.085649 1032 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:27.087469 1032 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:27.087563 1032 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:27.087708 1032 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:27.087785 1032 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:27.131994 1032 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.66:37795
I20260929 01:50:27.132025 1160 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.66:37795 every 8 connection(s)
I20260929 01:50:27.133569 1032 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
I20260929 01:50:27.134779 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 1032
I20260929 01:50:27.134989 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/wal/instance
I20260929 01:50:27.146100 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.67:0
--local_ip_for_outbound_sockets=127.0.71.67
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--time_source=system_unsync with env {}
I20260929 01:50:27.183336 1161 heartbeater.cc:344] Connected to a master server at 127.0.71.124:40033
I20260929 01:50:27.183720 1161 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:27.183832 1163 heartbeater.cc:344] Connected to a master server at 127.0.71.125:43325
I20260929 01:50:27.183991 1163 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:27.185070 1163 heartbeater.cc:507] Master 127.0.71.125:43325 requested a full tablet report, sending...
I20260929 01:50:27.185089 1161 heartbeater.cc:507] Master 127.0.71.124:40033 requested a full tablet report, sending...
I20260929 01:50:27.186796 734 ts_manager.cc:194] Registered new tserver with Master: a8b28722d60646f7adfa970805c79cf5 (127.0.71.66:37795)
I20260929 01:50:27.186885 804 ts_manager.cc:194] Registered new tserver with Master: a8b28722d60646f7adfa970805c79cf5 (127.0.71.66:37795)
I20260929 01:50:27.187867 734 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.66:42201
I20260929 01:50:27.189343 1162 heartbeater.cc:344] Connected to a master server at 127.0.71.126:42419
I20260929 01:50:27.189462 1162 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:27.189824 1162 heartbeater.cc:507] Master 127.0.71.126:42419 requested a full tablet report, sending...
I20260929 01:50:27.190790 663 ts_manager.cc:194] Registered new tserver with Master: a8b28722d60646f7adfa970805c79cf5 (127.0.71.66:37795)
W20260929 01:50:27.529038 1167 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:27.529352 1167 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:27.529423 1167 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:27.538465 1167 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:27.538637 1167 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.67
I20260929 01:50:27.549559 1167 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:27.551301 1167 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:27.553288 1167 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:27.564098 1174 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:27.564159 1173 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:27.564599 1176 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:27.565272 1167 server_base.cc:1061] running on GCE node
I20260929 01:50:27.565814 1167 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:50:27.565948 1167 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:50:27.566011 1167 hybrid_clock.cc:648] HybridClock initialized: now 1790646627565992 us; error 0 us; skew 500 ppm
I20260929 01:50:27.566711 1167 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:27.568974 1167 webserver.cc:533] Webserver started at http://127.0.71.67:35057/ using document root <none> and password file <none>
I20260929 01:50:27.569671 1167 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:27.569765 1167 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:27.570052 1167 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:50:27.572821 1167 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/data/instance:
uuid: "00f3a34e6e0b4a90ad8e70203a79b60d"
format_stamp: "Formatted at 2026-09-29 01:50:27 on dist-test-slave-vz23"
I20260929 01:50:27.573714 1167 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/wal/instance:
uuid: "00f3a34e6e0b4a90ad8e70203a79b60d"
format_stamp: "Formatted at 2026-09-29 01:50:27 on dist-test-slave-vz23"
I20260929 01:50:27.579360 1167 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 01:50:27.583055 1181 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:27.584731 1167 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 01:50:27.584910 1167 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/wal
uuid: "00f3a34e6e0b4a90ad8e70203a79b60d"
format_stamp: "Formatted at 2026-09-29 01:50:27 on dist-test-slave-vz23"
I20260929 01:50:27.585142 1167 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:27.616760 1167 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:27.617746 1167 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:27.618018 1167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:27.618973 1167 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:27.620695 1167 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:27.620817 1167 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:27.620951 1167 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:27.621042 1167 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:27.659933 1167 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.67:33739
I20260929 01:50:27.659981 1293 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.67:33739 every 8 connection(s)
I20260929 01:50:27.661639 1167 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
I20260929 01:50:27.665850 1026 heartbeater.cc:499] Master 127.0.71.125:43325 was elected leader, sending a full tablet report...
I20260929 01:50:27.671685 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 1167
I20260929 01:50:27.671900 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/wal/instance
I20260929 01:50:27.683827 1296 heartbeater.cc:344] Connected to a master server at 127.0.71.125:43325
I20260929 01:50:27.684108 1294 heartbeater.cc:344] Connected to a master server at 127.0.71.124:40033
I20260929 01:50:27.684130 1295 heartbeater.cc:344] Connected to a master server at 127.0.71.126:42419
I20260929 01:50:27.684360 1296 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:27.684362 1294 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:27.684365 1295 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:27.685572 1295 heartbeater.cc:507] Master 127.0.71.126:42419 requested a full tablet report, sending...
I20260929 01:50:27.685613 1296 heartbeater.cc:507] Master 127.0.71.125:43325 requested a full tablet report, sending...
I20260929 01:50:27.685658 1294 heartbeater.cc:507] Master 127.0.71.124:40033 requested a full tablet report, sending...
I20260929 01:50:27.687188 734 ts_manager.cc:194] Registered new tserver with Master: 00f3a34e6e0b4a90ad8e70203a79b60d (127.0.71.67:33739)
I20260929 01:50:27.687295 804 ts_manager.cc:194] Registered new tserver with Master: 00f3a34e6e0b4a90ad8e70203a79b60d (127.0.71.67:33739)
I20260929 01:50:27.687448 663 ts_manager.cc:194] Registered new tserver with Master: 00f3a34e6e0b4a90ad8e70203a79b60d (127.0.71.67:33739)
I20260929 01:50:27.688257 734 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.67:60421
I20260929 01:50:27.698450 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:50:27.706318 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:50:27.706550 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 634
I20260929 01:50:27.707477 694 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:27.814699 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 634
I20260929 01:50:27.831313 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:42419
--webserver_interface=127.0.71.126
--webserver_port=46793
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033 with env {}
W20260929 01:50:27.965734 718 consensus_peers.cc:597] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc -> Peer 0ea38b4fa64841e8a36df8bb12a20110 (127.0.71.126:42419): Couldn't send request to peer 0ea38b4fa64841e8a36df8bb12a20110. Status: Network error: Client connection negotiation failed: client connection to 127.0.71.126:42419: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260929 01:50:28.165699 1302 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:28.166066 1302 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:28.166196 1302 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:28.174563 1302 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:50:28.174700 1302 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:28.174808 1302 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:50:28.174891 1302 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:50:28.185722 1302 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:42419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=46793
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:28.187348 1302 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:28.189253 1302 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260929 01:50:28.190506 1163 heartbeater.cc:499] Master 127.0.71.125:43325 was elected leader, sending a full tablet report...
W20260929 01:50:28.192832 1162 heartbeater.cc:646] Failed to heartbeat to 127.0.71.126:42419 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.71.126:42419: connect: Connection refused (error 111)
W20260929 01:50:28.198187 1310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:28.198086 1311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:28.198375 1313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:28.199218 1302 server_base.cc:1061] running on GCE node
I20260929 01:50:28.199914 1302 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:50:28.200094 1302 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:50:28.200204 1302 hybrid_clock.cc:648] HybridClock initialized: now 1790646628200166 us; error 0 us; skew 500 ppm
I20260929 01:50:28.200585 1302 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:28.203353 1302 webserver.cc:533] Webserver started at http://127.0.71.126:46793/ using document root <none> and password file <none>
I20260929 01:50:28.204124 1302 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:28.204259 1302 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:28.209815 1302 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260929 01:50:28.212577 1318 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:28.214022 1302 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:50:28.214197 1302 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "0ea38b4fa64841e8a36df8bb12a20110"
format_stamp: "Formatted at 2026-09-29 01:50:24 on dist-test-slave-vz23"
I20260929 01:50:28.214900 1302 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:28.242741 1302 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:28.243757 1302 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:28.244059 1302 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:28.264971 1302 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:42419
I20260929 01:50:28.265163 1369 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:42419 every 8 connection(s)
I20260929 01:50:28.266606 1302 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:50:28.269097 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 1302
I20260929 01:50:28.269469 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 897
I20260929 01:50:28.275315 1370 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:50:28.279433 1019 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:28.279529 1370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: Bootstrap starting.
I20260929 01:50:28.303129 1370 log.cc:826] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: Log is configured to *not* fsync() on all Append() calls
I20260929 01:50:28.313409 1370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:50:28.314023 1370 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: Bootstrap complete.
I20260929 01:50:28.323849 1370 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } }
I20260929 01:50:28.324827 1370 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea38b4fa64841e8a36df8bb12a20110, State: Initialized, Role: FOLLOWER
I20260929 01:50:28.325654 1370 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } }
I20260929 01:50:28.326507 1376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:28.326771 1376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:28.327788 1370 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:50:28.338579 1387 catalog_manager.cc:1308] Loaded cluster ID: 8f5451ce215a49c0b44c8b9a8e05085a
I20260929 01:50:28.338749 1387 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: loading cluster ID for follower catalog manager: success
I20260929 01:50:28.342867 1387 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: acquiring CA information for follower catalog manager: success
I20260929 01:50:28.345543 1387 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:50:28.390648 993 sys_catalog.cc:455] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a067d22fc04447ed8804d60c167c0bfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:28.392833 993 sys_catalog.cc:458] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:28.401500 1376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [sys.catalog]: SysCatalogTable state changed. Reason: New leader a067d22fc04447ed8804d60c167c0bfc. Latest consensus state: current_term: 1 leader_uuid: "a067d22fc04447ed8804d60c167c0bfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:28.401760 1376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:28.419627 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 897
I20260929 01:50:28.436987 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:39199
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--webserver_port=38429
--webserver_interface=127.0.71.65
--time_source=system_unsync with env {}
I20260929 01:50:28.690546 1296 heartbeater.cc:499] Master 127.0.71.125:43325 was elected leader, sending a full tablet report...
I20260929 01:50:28.712138 1335 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "00f3a34e6e0b4a90ad8e70203a79b60d" instance_seqno: 1790646627650450) as {username='slave'} at 127.0.71.67:47297; Asking this server to re-register.
I20260929 01:50:28.713078 1295 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:28.715022 1335 ts_manager.cc:194] Registered new tserver with Master: 00f3a34e6e0b4a90ad8e70203a79b60d (127.0.71.67:33739)
W20260929 01:50:28.796494 1395 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:28.796864 1395 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:28.797009 1395 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:28.805456 1395 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:28.805683 1395 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:50:28.816016 1395 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:39199
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=38429
--tserver_master_addrs=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:28.817808 1395 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:28.819578 1395 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:28.829228 1402 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:28.829289 1401 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:28.830595 1404 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:28.831218 1395 server_base.cc:1061] running on GCE node
I20260929 01:50:28.831858 1395 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:50:28.832063 1395 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:50:28.832180 1395 hybrid_clock.cc:648] HybridClock initialized: now 1790646628832138 us; error 0 us; skew 500 ppm
I20260929 01:50:28.832588 1395 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:28.835062 1395 webserver.cc:533] Webserver started at http://127.0.71.65:38429/ using document root <none> and password file <none>
I20260929 01:50:28.835968 1395 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:28.836138 1395 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:28.842042 1395 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 01:50:28.844923 1409 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:28.846460 1395 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 01:50:28.846704 1395 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "9f86d7b4e82e43079d9a7d728aca3860"
format_stamp: "Formatted at 2026-09-29 01:50:26 on dist-test-slave-vz23"
I20260929 01:50:28.847422 1395 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:28.865155 1395 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:28.866058 1395 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:28.866393 1395 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:28.867421 1395 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:28.869277 1395 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:28.869431 1395 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:28.869592 1395 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:28.869668 1395 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:28.906083 1395 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:39199
I20260929 01:50:28.906136 1521 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:39199 every 8 connection(s)
I20260929 01:50:28.907636 1395 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:50:28.908733 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 1395
I20260929 01:50:28.909106 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 1395
I20260929 01:50:28.928730 993 sys_catalog.cc:455] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "a067d22fc04447ed8804d60c167c0bfc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:28.929239 993 sys_catalog.cc:458] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:28.944787 1522 heartbeater.cc:344] Connected to a master server at 127.0.71.124:40033
I20260929 01:50:28.945256 1522 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:28.945549 1517 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:29.040028 1522 heartbeater.cc:507] Master 127.0.71.124:40033 requested a full tablet report, sending...
I20260929 01:50:29.040616 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 1395
I20260929 01:50:29.059279 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:39199
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--webserver_port=38429
--webserver_interface=127.0.71.65
--time_source=system_unsync
--fault_before_start=1.0 with env {}
W20260929 01:50:29.059402 1534 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:50:28.932708 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:51443 (local address 127.0.71.125:43325)
0929 01:50:28.948973 (+ 16265us) server_negotiation.cc:207] Beginning negotiation
0929 01:50:28.948983 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0929 01:50:28.949008 (+ 25us) server_negotiation.cc:408] Connection header received
0929 01:50:28.949079 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:50:28.949088 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:50:28.949186 (+ 98us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:50:28.949330 (+ 144us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:50:29.042755 (+ 93425us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:50:29.043628 (+ 873us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:50:29.058355 (+ 14727us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:51443: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":16124,"thread_start_us":16010,"threads_started":1}
I20260929 01:50:29.201050 1162 heartbeater.cc:344] Connected to a master server at 127.0.71.126:42419
I20260929 01:50:29.202286 1335 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "a8b28722d60646f7adfa970805c79cf5" instance_seqno: 1790646627122256) as {username='slave'} at 127.0.71.66:42341; Asking this server to re-register.
I20260929 01:50:29.202888 1162 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:29.204005 1335 ts_manager.cc:194] Registered new tserver with Master: a8b28722d60646f7adfa970805c79cf5 (127.0.71.66:37795)
W20260929 01:50:29.401573 1537 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:29.401870 1537 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:29.402009 1537 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:50:29.402094 1537 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:29.410799 1537 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:29.411037 1537 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:50:29.421259 1537 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:39199
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=38429
--tserver_master_addrs=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:29.422875 1537 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:29.424654 1537 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:29.434392 1544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:29.434422 1543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:29.435248 1546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:29.435883 1537 server_base.cc:1061] running on GCE node
I20260929 01:50:29.436583 1537 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:50:29.436704 1537 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:50:29.436777 1537 hybrid_clock.cc:648] HybridClock initialized: now 1790646629436749 us; error 0 us; skew 500 ppm
I20260929 01:50:29.437002 1537 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:29.439229 1537 webserver.cc:533] Webserver started at http://127.0.71.65:38429/ using document root <none> and password file <none>
I20260929 01:50:29.439877 1537 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:29.440012 1537 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:29.445699 1537 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 01:50:29.448227 1551 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:29.449620 1537 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:50:29.449810 1537 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "9f86d7b4e82e43079d9a7d728aca3860"
format_stamp: "Formatted at 2026-09-29 01:50:26 on dist-test-slave-vz23"
I20260929 01:50:29.450698 1537 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:29.469290 1537 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:29.470155 1537 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:29.470427 1537 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:29.471374 1537 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:29.473094 1537 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:29.473217 1537 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:29.473353 1537 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:29.473441 1537 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:50:29.473801 1537 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:50:29.473922 1537 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:50:29.474025 1537 fault_injection.cc:59] injecting fault for kudu (pid 1537): FLAGS_fault_before_start (process will exit)
I20260929 01:50:29.497170 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:39199
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--webserver_port=38429
--webserver_interface=127.0.71.65
--time_source=system_unsync with env {}
W20260929 01:50:29.825255 1559 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:29.825687 1559 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:29.825855 1559 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:29.834515 1559 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:29.834688 1559 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:50:29.844549 1559 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:39199
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=38429
--tserver_master_addrs=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:29.846235 1559 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:29.848045 1559 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:29.857573 1564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:29.857611 1565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:29.858659 1567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:29.858991 1559 server_base.cc:1061] running on GCE node
I20260929 01:50:29.859629 1559 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:50:29.859817 1559 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:50:29.859908 1559 hybrid_clock.cc:648] HybridClock initialized: now 1790646629859888 us; error 0 us; skew 500 ppm
I20260929 01:50:29.860258 1559 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:29.862799 1559 webserver.cc:533] Webserver started at http://127.0.71.65:38429/ using document root <none> and password file <none>
I20260929 01:50:29.863643 1559 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:29.863767 1559 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:29.870121 1559 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:50:29.873208 1572 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:29.874809 1559 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:50:29.875025 1559 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "9f86d7b4e82e43079d9a7d728aca3860"
format_stamp: "Formatted at 2026-09-29 01:50:26 on dist-test-slave-vz23"
I20260929 01:50:29.875778 1559 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:29.897364 1559 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:29.898254 1559 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:29.898535 1559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:29.899452 1559 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:29.901064 1559 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:29.901167 1559 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:29.901312 1559 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:29.901382 1559 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:29.937184 1559 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:39199
I20260929 01:50:29.937233 1684 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:39199 every 8 connection(s)
I20260929 01:50:29.938716 1559 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:50:29.944788 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 1559
I20260929 01:50:29.945230 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 1559
I20260929 01:50:29.957051 1685 heartbeater.cc:344] Connected to a master server at 127.0.71.124:40033
I20260929 01:50:29.957506 1685 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:29.958303 1680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:29.958396 1685 heartbeater.cc:507] Master 127.0.71.124:40033 requested a full tablet report, sending...
I20260929 01:50:29.960438 804 ts_manager.cc:194] Re-registered known tserver with Master: 9f86d7b4e82e43079d9a7d728aca3860 (127.0.71.65:39199)
W20260929 01:50:30.044672 1575 outbound_call.cc:321] RPC callback for RPC call kudu.master.MasterService.TSHeartbeat -> {remote=127.0.71.124:40033, user_credentials={real_user=slave}} blocked reactor thread for 82557us
I20260929 01:50:30.046195 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 1559
W20260929 01:50:30.063297 1694 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:50:29.944204 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:43271 (local address 127.0.71.125:43325)
0929 01:50:29.944727 (+ 523us) server_negotiation.cc:207] Beginning negotiation
0929 01:50:29.944731 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0929 01:50:29.955403 (+ 10672us) server_negotiation.cc:408] Connection header received
0929 01:50:29.955864 (+ 461us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:50:29.955871 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:50:29.955950 (+ 79us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:50:29.956070 (+ 120us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:50:29.959426 (+ 3356us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:50:29.960332 (+ 906us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:50:30.062943 (+102611us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:43271: BlockingRecv error: recv got EOF from 127.0.71.65:43271 (error 108)
Metrics: {"server-negotiator.queue_time_us":431,"thread_start_us":337,"threads_started":1}
I20260929 01:50:30.063418 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 1032
W20260929 01:50:30.064074 1692 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:50:29.943570 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:46499 (local address 127.0.71.126:42419)
0929 01:50:29.965390 (+ 21820us) server_negotiation.cc:207] Beginning negotiation
0929 01:50:29.965395 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0929 01:50:29.965410 (+ 15us) server_negotiation.cc:408] Connection header received
0929 01:50:29.965467 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:50:29.965471 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:50:29.965529 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:50:29.965613 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:50:30.045663 (+ 80050us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:50:30.047145 (+ 1482us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:50:30.063103 (+ 15958us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:46499: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":21727,"thread_start_us":22137,"threads_started":1}
I20260929 01:50:30.071084 1156 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:30.157932 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 1032
I20260929 01:50:30.174115 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 1167
I20260929 01:50:30.181916 1289 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:30.275048 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 1167
I20260929 01:50:30.293452 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 1302
I20260929 01:50:30.300076 1365 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:30.404392 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 1302
I20260929 01:50:30.424166 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 701
I20260929 01:50:30.425446 764 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:30.531708 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 701
W20260929 01:50:30.551188 790 connection.cc:573] server connection from 127.0.0.1:45110 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:50:30.551553 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 771
I20260929 01:50:30.552752 834 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:30.658953 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 771
I20260929 01:50:30.683319 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:42419
--webserver_interface=127.0.71.126
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033 with env {}
W20260929 01:50:31.002774 1706 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:31.003155 1706 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:31.003293 1706 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:31.013964 1706 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:50:31.014087 1706 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:31.014176 1706 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:50:31.014262 1706 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:50:31.024271 1706 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:42419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:31.025923 1706 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:31.027690 1706 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:31.035450 1711 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:31.035879 1712 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:31.036218 1714 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:31.036724 1706 server_base.cc:1061] running on GCE node
I20260929 01:50:31.037706 1706 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:50:31.037859 1706 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:50:31.037925 1706 hybrid_clock.cc:648] HybridClock initialized: now 1790646631037909 us; error 0 us; skew 500 ppm
I20260929 01:50:31.038183 1706 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:31.040820 1706 webserver.cc:533] Webserver started at http://127.0.71.126:42363/ using document root <none> and password file <none>
I20260929 01:50:31.041468 1706 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:31.041554 1706 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:31.046787 1706 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 01:50:31.049540 1719 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:31.050904 1706 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:50:31.051119 1706 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "0ea38b4fa64841e8a36df8bb12a20110"
format_stamp: "Formatted at 2026-09-29 01:50:24 on dist-test-slave-vz23"
I20260929 01:50:31.051811 1706 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:31.089820 1706 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:31.090756 1706 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:31.091078 1706 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:31.111541 1706 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:42419
I20260929 01:50:31.111579 1770 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:42419 every 8 connection(s)
I20260929 01:50:31.113185 1706 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:50:31.120103 1771 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:50:31.120839 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 1706
I20260929 01:50:31.121104 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:50:31.122157 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:43325
--webserver_interface=127.0.71.125
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033 with env {}
I20260929 01:50:31.123522 1771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: Bootstrap starting.
I20260929 01:50:31.139443 1771 log.cc:826] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: Log is configured to *not* fsync() on all Append() calls
I20260929 01:50:31.147758 1771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:50:31.148272 1771 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: Bootstrap complete.
I20260929 01:50:31.154793 1771 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } }
I20260929 01:50:31.155457 1771 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea38b4fa64841e8a36df8bb12a20110, State: Initialized, Role: FOLLOWER
I20260929 01:50:31.156174 1771 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } }
I20260929 01:50:31.157131 1775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:31.157413 1775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:31.158109 1771 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:50:31.172248 1786 catalog_manager.cc:1308] Loaded cluster ID: 8f5451ce215a49c0b44c8b9a8e05085a
I20260929 01:50:31.172515 1786 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: loading cluster ID for follower catalog manager: success
I20260929 01:50:31.178282 1786 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: acquiring CA information for follower catalog manager: success
I20260929 01:50:31.181723 1786 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:50:31.193446 1706 master_runner.cc:187] Error getting master registration for 127.0.71.125:43325: Network error: Client connection negotiation failed: client connection to 127.0.71.125:43325: connect: Connection refused (error 111),
I20260929 01:50:31.197078 1706 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:50:31.472788 1773 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:31.473160 1773 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:31.473284 1773 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:31.481832 1773 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:50:31.481971 1773 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:31.482086 1773 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:50:31.482170 1773 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:50:31.492353 1773 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:43325
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:31.494000 1773 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:31.495811 1773 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:31.503963 1794 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:31.504132 1795 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:31.505560 1797 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:31.506254 1773 server_base.cc:1061] running on GCE node
I20260929 01:50:31.507062 1773 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:50:31.507267 1773 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:50:31.507380 1773 hybrid_clock.cc:648] HybridClock initialized: now 1790646631507358 us; error 0 us; skew 500 ppm
I20260929 01:50:31.507762 1773 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:31.510792 1773 webserver.cc:533] Webserver started at http://127.0.71.125:41419/ using document root <none> and password file <none>
I20260929 01:50:31.511551 1773 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:31.511686 1773 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:31.517614 1773 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:50:31.520888 1802 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:31.522385 1773 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 01:50:31.522576 1773 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "a067d22fc04447ed8804d60c167c0bfc"
format_stamp: "Formatted at 2026-09-29 01:50:25 on dist-test-slave-vz23"
I20260929 01:50:31.523360 1773 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:31.543355 1773 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:31.544286 1773 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:31.544641 1773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:31.567047 1773 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:43325
I20260929 01:50:31.567047 1853 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:43325 every 8 connection(s)
I20260929 01:50:31.568805 1773 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:50:31.569259 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 1773
I20260929 01:50:31.569465 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:50:31.570717 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:40033
--webserver_interface=127.0.71.124
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033 with env {}
I20260929 01:50:31.579066 1854 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:50:31.582643 1854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc: Bootstrap starting.
I20260929 01:50:31.598866 1854 log.cc:826] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc: Log is configured to *not* fsync() on all Append() calls
I20260929 01:50:31.607232 1854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:50:31.607774 1854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc: Bootstrap complete.
I20260929 01:50:31.614372 1854 raft_consensus.cc:359] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } }
I20260929 01:50:31.615077 1854 raft_consensus.cc:740] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: a067d22fc04447ed8804d60c167c0bfc, State: Initialized, Role: FOLLOWER
I20260929 01:50:31.615808 1854 consensus_queue.cc:260] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } }
I20260929 01:50:31.616667 1858 sys_catalog.cc:455] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:31.616950 1858 sys_catalog.cc:458] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:31.617691 1854 sys_catalog.cc:565] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:50:31.627938 1869 catalog_manager.cc:1308] Loaded cluster ID: 8f5451ce215a49c0b44c8b9a8e05085a
I20260929 01:50:31.628180 1869 catalog_manager.cc:1601] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc: loading cluster ID for follower catalog manager: success
I20260929 01:50:31.633453 1869 catalog_manager.cc:1623] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc: acquiring CA information for follower catalog manager: success
I20260929 01:50:31.636181 1869 catalog_manager.cc:1651] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:50:31.657013 1773 master_runner.cc:187] Error getting master registration for 127.0.71.124:40033: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40033: connect: Connection refused (error 111),
I20260929 01:50:31.661032 1773 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:50:31.927768 1856 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:31.928145 1856 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:31.928279 1856 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:31.936839 1856 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:50:31.936971 1856 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:31.937101 1856 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:50:31.937186 1856 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:50:31.947296 1856 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:40033
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:31.949081 1856 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:31.950891 1856 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:31.959303 1878 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:31.960352 1877 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:31.961438 1856 server_base.cc:1061] running on GCE node
W20260929 01:50:31.961454 1880 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:31.962316 1856 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:50:31.962503 1856 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:50:31.962605 1856 hybrid_clock.cc:648] HybridClock initialized: now 1790646631962584 us; error 0 us; skew 500 ppm
I20260929 01:50:31.962947 1856 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:31.965826 1856 webserver.cc:533] Webserver started at http://127.0.71.124:36341/ using document root <none> and password file <none>
I20260929 01:50:31.966558 1856 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:31.966702 1856 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:31.972260 1856 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 01:50:31.975200 1885 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:31.976783 1856 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:50:31.976958 1856 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "96a59e909fb44ae1bc1b7ac327df09b5"
format_stamp: "Formatted at 2026-09-29 01:50:25 on dist-test-slave-vz23"
I20260929 01:50:31.977684 1856 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:31.996176 1856 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:31.997174 1856 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:31.997483 1856 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:32.018754 1856 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:40033
I20260929 01:50:32.018771 1936 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:40033 every 8 connection(s)
I20260929 01:50:32.020666 1856 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:50:32.027773 1937 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:50:32.030333 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 1856
I20260929 01:50:32.030522 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:50:32.031188 1937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5: Bootstrap starting.
I20260929 01:50:32.050493 1937 log.cc:826] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5: Log is configured to *not* fsync() on all Append() calls
I20260929 01:50:32.059999 1937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:50:32.060729 1937 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5: Bootstrap complete.
I20260929 01:50:32.067162 1937 raft_consensus.cc:359] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } }
I20260929 01:50:32.067855 1937 raft_consensus.cc:740] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 96a59e909fb44ae1bc1b7ac327df09b5, State: Initialized, Role: FOLLOWER
I20260929 01:50:32.068575 1937 consensus_queue.cc:260] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } }
I20260929 01:50:32.069363 1942 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:32.069823 1942 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:32.070473 1937 sys_catalog.cc:565] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:50:32.081432 1953 catalog_manager.cc:1308] Loaded cluster ID: 8f5451ce215a49c0b44c8b9a8e05085a
I20260929 01:50:32.081554 1953 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5: loading cluster ID for follower catalog manager: success
I20260929 01:50:32.086087 1953 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5: acquiring CA information for follower catalog manager: success
I20260929 01:50:32.088896 1953 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:50:32.109383 1856 master_runner.cc:264] No leader master found from master 96a59e909fb44ae1bc1b7ac327df09b5
I20260929 01:50:32.109582 1856 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:50:32.788038 1956 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:50:32.788280 1956 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } }
I20260929 01:50:32.790172 1956 leader_election.cc:290] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers a067d22fc04447ed8804d60c167c0bfc (127.0.71.125:43325), 96a59e909fb44ae1bc1b7ac327df09b5 (127.0.71.124:40033)
I20260929 01:50:32.798795 1912 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ea38b4fa64841e8a36df8bb12a20110" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" is_pre_election: true
I20260929 01:50:32.799068 1829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ea38b4fa64841e8a36df8bb12a20110" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a067d22fc04447ed8804d60c167c0bfc" is_pre_election: true
I20260929 01:50:32.799374 1912 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ea38b4fa64841e8a36df8bb12a20110 in term 1.
I20260929 01:50:32.799540 1829 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0ea38b4fa64841e8a36df8bb12a20110 in term 1.
I20260929 01:50:32.800006 1722 leader_election.cc:304] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea38b4fa64841e8a36df8bb12a20110, 96a59e909fb44ae1bc1b7ac327df09b5; no voters:
I20260929 01:50:32.800365 1956 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 01:50:32.800580 1956 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:50:32.800657 1956 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:50:32.804057 1956 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } }
I20260929 01:50:32.804718 1956 leader_election.cc:290] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [CANDIDATE]: Term 2 election: Requested vote from peers a067d22fc04447ed8804d60c167c0bfc (127.0.71.125:43325), 96a59e909fb44ae1bc1b7ac327df09b5 (127.0.71.124:40033)
I20260929 01:50:32.805065 1829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ea38b4fa64841e8a36df8bb12a20110" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "a067d22fc04447ed8804d60c167c0bfc"
I20260929 01:50:32.805269 1829 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:50:32.805269 1912 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0ea38b4fa64841e8a36df8bb12a20110" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "96a59e909fb44ae1bc1b7ac327df09b5"
I20260929 01:50:32.805459 1912 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:50:32.808787 1829 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ea38b4fa64841e8a36df8bb12a20110 in term 2.
I20260929 01:50:32.808887 1912 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0ea38b4fa64841e8a36df8bb12a20110 in term 2.
I20260929 01:50:32.809253 1720 leader_election.cc:304] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea38b4fa64841e8a36df8bb12a20110, a067d22fc04447ed8804d60c167c0bfc; no voters:
I20260929 01:50:32.809564 1956 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:50:32.809966 1956 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [term 2 LEADER]: Becoming Leader. State: Replica: 0ea38b4fa64841e8a36df8bb12a20110, State: Running, Role: LEADER
I20260929 01:50:32.810412 1956 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } }
I20260929 01:50:32.812875 1961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ea38b4fa64841e8a36df8bb12a20110. Latest consensus state: current_term: 2 leader_uuid: "0ea38b4fa64841e8a36df8bb12a20110" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:32.813107 1961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:32.813690 1963 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:50:32.815534 1963 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:50:32.816418 1963 catalog_manager.cc:1308] Loaded cluster ID: 8f5451ce215a49c0b44c8b9a8e05085a
I20260929 01:50:32.816654 1963 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:50:32.817745 1963 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:50:32.818540 1963 catalog_manager.cc:6230] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: Loaded TSK: 0
I20260929 01:50:32.819412 1963 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:50:32.837258 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--time_source=system_unsync with env {}
W20260929 01:50:33.167092 1964 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:33.167460 1964 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:33.167593 1964 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:33.176002 1964 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:33.176223 1964 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:50:33.186100 1964 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:33.187662 1964 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:33.189606 1964 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:33.199347 1970 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:33.199617 1969 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:33.199801 1972 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:33.200825 1964 server_base.cc:1061] running on GCE node
I20260929 01:50:33.201423 1964 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:50:33.201594 1964 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:50:33.201675 1964 hybrid_clock.cc:648] HybridClock initialized: now 1790646633201658 us; error 0 us; skew 500 ppm
I20260929 01:50:33.201977 1964 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:33.204210 1964 webserver.cc:533] Webserver started at http://127.0.71.65:44003/ using document root <none> and password file <none>
I20260929 01:50:33.205024 1964 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:33.205117 1964 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:33.211133 1964 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 01:50:33.213994 1977 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:33.215348 1964 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:50:33.215526 1964 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "9f86d7b4e82e43079d9a7d728aca3860"
format_stamp: "Formatted at 2026-09-29 01:50:26 on dist-test-slave-vz23"
I20260929 01:50:33.216285 1964 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:33.231240 1964 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:33.232086 1964 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:33.232376 1964 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:33.233356 1964 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:33.234973 1964 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:33.235086 1964 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:33.235234 1964 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:33.235304 1964 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:33.271091 1964 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:35189
I20260929 01:50:33.271239 2089 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:35189 every 8 connection(s)
I20260929 01:50:33.272672 1964 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:50:33.275082 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 1964
I20260929 01:50:33.275319 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
I20260929 01:50:33.280063 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.66:0
--local_ip_for_outbound_sockets=127.0.71.66
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--time_source=system_unsync with env {}
I20260929 01:50:33.289716 1912 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [term 2 FOLLOWER]: Refusing update from remote peer 0ea38b4fa64841e8a36df8bb12a20110: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 01:50:33.290940 1961 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [LEADER]: Connected to new peer: Peer: permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:50:33.300370 2091 heartbeater.cc:344] Connected to a master server at 127.0.71.126:42419
I20260929 01:50:33.300379 2090 heartbeater.cc:344] Connected to a master server at 127.0.71.124:40033
I20260929 01:50:33.300769 2091 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:33.300774 2090 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:33.302114 2090 heartbeater.cc:507] Master 127.0.71.124:40033 requested a full tablet report, sending...
I20260929 01:50:33.303229 2092 heartbeater.cc:344] Connected to a master server at 127.0.71.125:43325
I20260929 01:50:33.303361 2092 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:33.303920 2092 heartbeater.cc:507] Master 127.0.71.125:43325 requested a full tablet report, sending...
I20260929 01:50:33.304422 2091 heartbeater.cc:507] Master 127.0.71.126:42419 requested a full tablet report, sending...
I20260929 01:50:33.306025 1818 ts_manager.cc:194] Registered new tserver with Master: 9f86d7b4e82e43079d9a7d728aca3860 (127.0.71.65:35189)
I20260929 01:50:33.308023 1736 ts_manager.cc:194] Registered new tserver with Master: 9f86d7b4e82e43079d9a7d728aca3860 (127.0.71.65:35189)
I20260929 01:50:33.310401 1736 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:42447
I20260929 01:50:33.318176 2099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ea38b4fa64841e8a36df8bb12a20110. Latest consensus state: current_term: 2 leader_uuid: "0ea38b4fa64841e8a36df8bb12a20110" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:33.318792 2099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:33.318881 1902 ts_manager.cc:194] Registered new tserver with Master: 9f86d7b4e82e43079d9a7d728aca3860 (127.0.71.65:35189)
I20260929 01:50:33.320819 1961 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "0ea38b4fa64841e8a36df8bb12a20110" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:33.322980 1829 raft_consensus.cc:1275] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [term 2 FOLLOWER]: Refusing update from remote peer 0ea38b4fa64841e8a36df8bb12a20110: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 01:50:33.324316 1961 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:33.331250 1961 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:50:33.350827 2099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "0ea38b4fa64841e8a36df8bb12a20110" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:33.351076 2099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:33.352131 2109 sys_catalog.cc:455] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "0ea38b4fa64841e8a36df8bb12a20110" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:33.352550 2109 sys_catalog.cc:458] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:33.352877 2106 sys_catalog.cc:455] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0ea38b4fa64841e8a36df8bb12a20110. Latest consensus state: current_term: 2 leader_uuid: "0ea38b4fa64841e8a36df8bb12a20110" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:33.353127 2106 sys_catalog.cc:458] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:50:33.644321 2097 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:33.644694 2097 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:33.644832 2097 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:33.654618 2097 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:33.654843 2097 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.66
I20260929 01:50:33.665138 2097 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:33.666756 2097 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:33.668542 2097 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:33.677992 2114 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:33.678122 2115 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:33.681900 2117 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:33.682772 2097 server_base.cc:1061] running on GCE node
I20260929 01:50:33.683430 2097 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:50:33.683655 2097 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:50:33.683754 2097 hybrid_clock.cc:648] HybridClock initialized: now 1790646633683716 us; error 0 us; skew 500 ppm
I20260929 01:50:33.684127 2097 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:33.691157 2097 webserver.cc:533] Webserver started at http://127.0.71.66:32991/ using document root <none> and password file <none>
I20260929 01:50:33.691944 2097 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:33.692085 2097 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:33.697563 2097 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:50:33.700203 2122 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:33.701625 2097 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:50:33.701824 2097 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/wal
uuid: "a8b28722d60646f7adfa970805c79cf5"
format_stamp: "Formatted at 2026-09-29 01:50:27 on dist-test-slave-vz23"
I20260929 01:50:33.702558 2097 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:33.714349 2097 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:33.715276 2097 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:33.715595 2097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:33.716593 2097 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:33.718333 2097 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:33.718461 2097 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:33.718645 2097 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:33.718724 2097 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:33.756430 2097 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.66:33285
I20260929 01:50:33.756518 2234 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.66:33285 every 8 connection(s)
I20260929 01:50:33.758047 2097 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
I20260929 01:50:33.759932 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 2097
I20260929 01:50:33.760119 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-1/wal/instance
I20260929 01:50:33.769652 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.67:0
--local_ip_for_outbound_sockets=127.0.71.67
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--time_source=system_unsync with env {}
I20260929 01:50:33.784036 2101 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "0ea38b4fa64841e8a36df8bb12a20110" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:33.784317 2101 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:33.798334 2237 heartbeater.cc:344] Connected to a master server at 127.0.71.125:43325
I20260929 01:50:33.798650 2235 heartbeater.cc:344] Connected to a master server at 127.0.71.124:40033
I20260929 01:50:33.798928 2235 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:33.799033 2237 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:33.800053 2237 heartbeater.cc:507] Master 127.0.71.125:43325 requested a full tablet report, sending...
I20260929 01:50:33.800139 2235 heartbeater.cc:507] Master 127.0.71.124:40033 requested a full tablet report, sending...
I20260929 01:50:33.801991 1902 ts_manager.cc:194] Registered new tserver with Master: a8b28722d60646f7adfa970805c79cf5 (127.0.71.66:33285)
I20260929 01:50:33.802095 1818 ts_manager.cc:194] Registered new tserver with Master: a8b28722d60646f7adfa970805c79cf5 (127.0.71.66:33285)
I20260929 01:50:33.804973 2236 heartbeater.cc:344] Connected to a master server at 127.0.71.126:42419
I20260929 01:50:33.805117 2236 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:33.805696 2236 heartbeater.cc:507] Master 127.0.71.126:42419 requested a full tablet report, sending...
I20260929 01:50:33.806881 1736 ts_manager.cc:194] Registered new tserver with Master: a8b28722d60646f7adfa970805c79cf5 (127.0.71.66:33285)
I20260929 01:50:33.808043 1736 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.66:33113
W20260929 01:50:34.126235 2241 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:34.126600 2241 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:34.126731 2241 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:34.135154 2241 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:34.135409 2241 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.67
I20260929 01:50:34.145085 2241 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:34.146689 2241 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:34.148424 2241 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:34.158030 2248 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:34.158092 2247 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:34.158088 2250 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:34.159103 2241 server_base.cc:1061] running on GCE node
I20260929 01:50:34.159668 2241 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:50:34.159857 2241 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:50:34.160032 2241 hybrid_clock.cc:648] HybridClock initialized: now 1790646634159938 us; error 0 us; skew 500 ppm
I20260929 01:50:34.160336 2241 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:34.162837 2241 webserver.cc:533] Webserver started at http://127.0.71.67:33241/ using document root <none> and password file <none>
I20260929 01:50:34.163589 2241 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:34.163753 2241 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:34.169330 2241 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:50:34.172379 2255 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:34.173974 2241 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:50:34.174145 2241 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/wal
uuid: "00f3a34e6e0b4a90ad8e70203a79b60d"
format_stamp: "Formatted at 2026-09-29 01:50:27 on dist-test-slave-vz23"
I20260929 01:50:34.174839 2241 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:34.219616 2241 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:34.220716 2241 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:34.221086 2241 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:34.222189 2241 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:34.223812 2241 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:34.223922 2241 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:34.224071 2241 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:34.224141 2241 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:34.258364 2241 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.67:34969
I20260929 01:50:34.258457 2367 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.67:34969 every 8 connection(s)
I20260929 01:50:34.259883 2241 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
I20260929 01:50:34.262825 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 2241
I20260929 01:50:34.263110 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-2/wal/instance
I20260929 01:50:34.284749 2370 heartbeater.cc:344] Connected to a master server at 127.0.71.125:43325
I20260929 01:50:34.284796 2368 heartbeater.cc:344] Connected to a master server at 127.0.71.124:40033
I20260929 01:50:34.285300 2369 heartbeater.cc:344] Connected to a master server at 127.0.71.126:42419
I20260929 01:50:34.285406 2368 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:34.285449 2369 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:34.285403 2370 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:34.286516 2369 heartbeater.cc:507] Master 127.0.71.126:42419 requested a full tablet report, sending...
I20260929 01:50:34.286553 2370 heartbeater.cc:507] Master 127.0.71.125:43325 requested a full tablet report, sending...
I20260929 01:50:34.286607 2368 heartbeater.cc:507] Master 127.0.71.124:40033 requested a full tablet report, sending...
I20260929 01:50:34.288384 1736 ts_manager.cc:194] Registered new tserver with Master: 00f3a34e6e0b4a90ad8e70203a79b60d (127.0.71.67:34969)
I20260929 01:50:34.288393 1818 ts_manager.cc:194] Registered new tserver with Master: 00f3a34e6e0b4a90ad8e70203a79b60d (127.0.71.67:34969)
I20260929 01:50:34.288511 1902 ts_manager.cc:194] Registered new tserver with Master: 00f3a34e6e0b4a90ad8e70203a79b60d (127.0.71.67:34969)
I20260929 01:50:34.289690 1736 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.67:38953
I20260929 01:50:34.294857 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:50:34.303021 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:50:34.303253 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 1706
I20260929 01:50:34.304340 1766 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:34.314069 2091 heartbeater.cc:499] Master 127.0.71.126:42419 was elected leader, sending a full tablet report...
I20260929 01:50:34.414301 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 1706
I20260929 01:50:34.435376 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:42419
--webserver_interface=127.0.71.126
--webserver_port=42363
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033 with env {}
W20260929 01:50:34.765758 2376 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:34.766052 2376 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:34.766122 2376 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:34.774670 2376 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:50:34.774760 2376 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:34.774813 2376 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:50:34.774854 2376 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:50:34.785262 2376 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:42419
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=42363
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:34.786778 2376 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:34.788431 2376 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:34.796742 2382 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:34.796906 2384 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:34.797372 2381 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:34.797689 2376 server_base.cc:1061] running on GCE node
I20260929 01:50:34.798311 2376 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:50:34.798444 2376 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:50:34.798506 2376 hybrid_clock.cc:648] HybridClock initialized: now 1790646634798489 us; error 0 us; skew 500 ppm
I20260929 01:50:34.798741 2376 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:34.801436 2376 webserver.cc:533] Webserver started at http://127.0.71.126:42363/ using document root <none> and password file <none>
I20260929 01:50:34.802083 2376 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:34.802169 2376 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:34.808032 2376 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 01:50:34.810729 2236 heartbeater.cc:499] Master 127.0.71.126:42419 was elected leader, sending a full tablet report...
I20260929 01:50:34.810909 2389 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20260929 01:50:34.812299 2236 heartbeater.cc:646] Failed to heartbeat to 127.0.71.126:42419 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.71.126:42419: connect: Connection refused (error 111)
I20260929 01:50:34.812393 2376 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 01:50:34.812584 2376 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "0ea38b4fa64841e8a36df8bb12a20110"
format_stamp: "Formatted at 2026-09-29 01:50:24 on dist-test-slave-vz23"
I20260929 01:50:34.813266 2376 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:34.846272 2376 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:34.847283 2376 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:34.847590 2376 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:34.868182 2376 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:42419
I20260929 01:50:34.868228 2441 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:42419 every 8 connection(s)
I20260929 01:50:34.869897 2376 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:50:34.872714 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 2376
I20260929 01:50:34.873075 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 1964
I20260929 01:50:34.878502 2442 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:50:34.882814 2085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:34.883068 2442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: Bootstrap starting.
I20260929 01:50:34.901846 2442 log.cc:826] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: Log is configured to *not* fsync() on all Append() calls
I20260929 01:50:34.911044 2442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:50:34.911590 2442 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: Bootstrap complete.
I20260929 01:50:34.918494 2442 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } }
I20260929 01:50:34.919468 2442 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0ea38b4fa64841e8a36df8bb12a20110, State: Initialized, Role: FOLLOWER
I20260929 01:50:34.920342 2442 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } }
I20260929 01:50:34.921419 2448 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:34.921761 2448 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:34.922618 2442 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:50:34.939901 2459 catalog_manager.cc:1308] Loaded cluster ID: 8f5451ce215a49c0b44c8b9a8e05085a
I20260929 01:50:34.940090 2459 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: loading cluster ID for follower catalog manager: success
I20260929 01:50:34.944177 2459 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: acquiring CA information for follower catalog manager: success
I20260929 01:50:34.947212 2459 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:50:34.974476 2376 master_runner.cc:264] No leader master found from master 0ea38b4fa64841e8a36df8bb12a20110
I20260929 01:50:34.974663 2376 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:50:34.984928 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 1964
I20260929 01:50:35.011876 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:35189
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--webserver_port=44003
--webserver_interface=127.0.71.65
--time_source=system_unsync with env {}
I20260929 01:50:35.292321 2369 heartbeater.cc:499] Master 127.0.71.126:42419 was elected leader, sending a full tablet report...
I20260929 01:50:35.303328 2407 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "00f3a34e6e0b4a90ad8e70203a79b60d" instance_seqno: 1790646634250015) as {username='slave'} at 127.0.71.67:43943; Asking this server to re-register.
I20260929 01:50:35.304260 2369 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:35.306066 2407 ts_manager.cc:194] Registered new tserver with Master: 00f3a34e6e0b4a90ad8e70203a79b60d (127.0.71.67:34969)
W20260929 01:50:35.358322 2465 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:35.358690 2465 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:35.358824 2465 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:35.367180 2465 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:35.367367 2465 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:50:35.377482 2465 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:35189
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=44003
--tserver_master_addrs=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:35.379192 2465 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:35.381115 2465 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:35.391433 2471 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:35.391619 2472 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:35.392194 2474 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:35.393095 2465 server_base.cc:1061] running on GCE node
I20260929 01:50:35.393646 2465 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:50:35.393857 2465 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:50:35.393939 2465 hybrid_clock.cc:648] HybridClock initialized: now 1790646635393921 us; error 0 us; skew 500 ppm
I20260929 01:50:35.394304 2465 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:35.396662 2465 webserver.cc:533] Webserver started at http://127.0.71.65:44003/ using document root <none> and password file <none>
I20260929 01:50:35.397420 2465 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:35.397516 2465 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:35.403163 2465 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:50:35.405889 2479 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:35.407449 2465 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:50:35.407668 2465 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "9f86d7b4e82e43079d9a7d728aca3860"
format_stamp: "Formatted at 2026-09-29 01:50:26 on dist-test-slave-vz23"
I20260929 01:50:35.408576 2465 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:35.423245 2465 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:35.424222 2465 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:35.424595 2465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:35.425566 2465 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:35.427452 2465 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:35.427559 2465 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:35.427722 2465 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:35.427799 2465 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:35.465724 2465 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:35189
I20260929 01:50:35.465758 2591 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:35189 every 8 connection(s)
I20260929 01:50:35.467371 2465 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:50:35.472827 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 2465
I20260929 01:50:35.473227 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 2465
I20260929 01:50:35.492202 2592 heartbeater.cc:344] Connected to a master server at 127.0.71.124:40033
I20260929 01:50:35.492596 2594 heartbeater.cc:344] Connected to a master server at 127.0.71.125:43325
I20260929 01:50:35.492712 2592 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:35.492776 2594 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:35.494012 2594 heartbeater.cc:507] Master 127.0.71.125:43325 requested a full tablet report, sending...
I20260929 01:50:35.494045 2593 heartbeater.cc:344] Connected to a master server at 127.0.71.126:42419
I20260929 01:50:35.494261 2593 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:35.494748 2593 heartbeater.cc:507] Master 127.0.71.126:42419 requested a full tablet report, sending...
I20260929 01:50:35.493999 2592 heartbeater.cc:507] Master 127.0.71.124:40033 requested a full tablet report, sending...
I20260929 01:50:35.496135 1818 ts_manager.cc:194] Re-registered known tserver with Master: 9f86d7b4e82e43079d9a7d728aca3860 (127.0.71.65:35189)
I20260929 01:50:35.496237 1901 ts_manager.cc:194] Re-registered known tserver with Master: 9f86d7b4e82e43079d9a7d728aca3860 (127.0.71.65:35189)
I20260929 01:50:35.497073 2407 ts_manager.cc:194] Registered new tserver with Master: 9f86d7b4e82e43079d9a7d728aca3860 (127.0.71.65:35189)
I20260929 01:50:35.501042 2587 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:35.587613 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 2465
I20260929 01:50:35.604079 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:35189
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--webserver_port=44003
--webserver_interface=127.0.71.65
--time_source=system_unsync
--fault_before_start=1.0 with env {}
I20260929 01:50:35.709430 2604 raft_consensus.cc:493] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ea38b4fa64841e8a36df8bb12a20110)
I20260929 01:50:35.709702 2604 raft_consensus.cc:515] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } }
I20260929 01:50:35.711606 2604 leader_election.cc:290] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0ea38b4fa64841e8a36df8bb12a20110 (127.0.71.126:42419), a067d22fc04447ed8804d60c167c0bfc (127.0.71.125:43325)
I20260929 01:50:35.712980 1829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a067d22fc04447ed8804d60c167c0bfc" is_pre_election: true
I20260929 01:50:35.718184 2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ea38b4fa64841e8a36df8bb12a20110" is_pre_election: true
I20260929 01:50:35.718667 2417 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 96a59e909fb44ae1bc1b7ac327df09b5 in term 2.
I20260929 01:50:35.719304 1886 leader_election.cc:304] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0ea38b4fa64841e8a36df8bb12a20110, 96a59e909fb44ae1bc1b7ac327df09b5; no voters: a067d22fc04447ed8804d60c167c0bfc
I20260929 01:50:35.719699 2604 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 01:50:35.719870 2604 raft_consensus.cc:493] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0ea38b4fa64841e8a36df8bb12a20110)
I20260929 01:50:35.719972 2604 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:50:35.721102 2606 raft_consensus.cc:493] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0ea38b4fa64841e8a36df8bb12a20110)
I20260929 01:50:35.721302 2606 raft_consensus.cc:515] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } }
I20260929 01:50:35.722942 2604 raft_consensus.cc:515] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } }
I20260929 01:50:35.724406 2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ea38b4fa64841e8a36df8bb12a20110"
I20260929 01:50:35.724663 2417 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:50:35.729735 2606 leader_election.cc:290] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0ea38b4fa64841e8a36df8bb12a20110 (127.0.71.126:42419), 96a59e909fb44ae1bc1b7ac327df09b5 (127.0.71.124:40033)
I20260929 01:50:35.730964 2417 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96a59e909fb44ae1bc1b7ac327df09b5 in term 3.
I20260929 01:50:35.731535 1886 leader_election.cc:304] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0ea38b4fa64841e8a36df8bb12a20110, 96a59e909fb44ae1bc1b7ac327df09b5; no voters:
I20260929 01:50:35.737573 2604 leader_election.cc:290] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [CANDIDATE]: Term 3 election: Requested vote from peers 0ea38b4fa64841e8a36df8bb12a20110 (127.0.71.126:42419), a067d22fc04447ed8804d60c167c0bfc (127.0.71.125:43325)
I20260929 01:50:35.738319 2604 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 01:50:35.738826 2604 raft_consensus.cc:697] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [term 3 LEADER]: Becoming Leader. State: Replica: 96a59e909fb44ae1bc1b7ac327df09b5, State: Running, Role: LEADER
I20260929 01:50:35.739472 2604 consensus_queue.cc:237] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } }
I20260929 01:50:35.743209 2417 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a067d22fc04447ed8804d60c167c0bfc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0ea38b4fa64841e8a36df8bb12a20110" is_pre_election: true
I20260929 01:50:35.743213 1911 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "a067d22fc04447ed8804d60c167c0bfc" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" is_pre_election: true
I20260929 01:50:35.743564 2417 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate a067d22fc04447ed8804d60c167c0bfc in current term 3: Already voted for candidate 96a59e909fb44ae1bc1b7ac327df09b5 in this term.
I20260929 01:50:35.744292 1803 leader_election.cc:304] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: a067d22fc04447ed8804d60c167c0bfc; no voters: 0ea38b4fa64841e8a36df8bb12a20110, 96a59e909fb44ae1bc1b7ac327df09b5
I20260929 01:50:35.745078 2606 raft_consensus.cc:3060] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:50:35.745071 2604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96a59e909fb44ae1bc1b7ac327df09b5. Latest consensus state: current_term: 3 leader_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:35.745306 2604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:35.746994 1829 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "a067d22fc04447ed8804d60c167c0bfc"
I20260929 01:50:35.747107 2611 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:50:35.747732 2606 raft_consensus.cc:2749] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 01:50:35.749764 2611 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:50:35.750506 1829 raft_consensus.cc:2468] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 96a59e909fb44ae1bc1b7ac327df09b5 in term 3.
I20260929 01:50:35.750751 2611 catalog_manager.cc:1308] Loaded cluster ID: 8f5451ce215a49c0b44c8b9a8e05085a
I20260929 01:50:35.750836 2611 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:50:35.752305 2611 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:50:35.753468 2611 catalog_manager.cc:6230] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5: Loaded TSK: 0
I20260929 01:50:35.754479 2611 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:50:35.819439 2236 heartbeater.cc:344] Connected to a master server at 127.0.71.126:42419
I20260929 01:50:35.819674 2236 heartbeater.cc:499] Master 127.0.71.126:42419 was elected leader, sending a full tablet report...
I20260929 01:50:35.820835 2407 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "a8b28722d60646f7adfa970805c79cf5" instance_seqno: 1790646633747634) as {username='slave'} at 127.0.71.66:49369; Asking this server to re-register.
I20260929 01:50:35.821557 2236 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:35.822700 2407 ts_manager.cc:194] Registered new tserver with Master: a8b28722d60646f7adfa970805c79cf5 (127.0.71.66:33285)
W20260929 01:50:35.947239 2603 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:35.947598 2603 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:35.947739 2603 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:50:35.947831 2603 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:35.956197 2603 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:35.956494 2603 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:50:35.966363 2603 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:35189
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=44003
--tserver_master_addrs=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:35.968031 2603 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:35.969919 2603 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:35.979369 2617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:35.979346 2618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:35.979890 2620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:35.980858 2603 server_base.cc:1061] running on GCE node
I20260929 01:50:35.981433 2603 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:50:35.981608 2603 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:50:35.981729 2603 hybrid_clock.cc:648] HybridClock initialized: now 1790646635981671 us; error 0 us; skew 500 ppm
I20260929 01:50:35.982074 2603 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:35.985886 2603 webserver.cc:533] Webserver started at http://127.0.71.65:44003/ using document root <none> and password file <none>
I20260929 01:50:35.986627 2603 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:35.986759 2603 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:35.992074 2603 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:50:35.994850 2625 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:35.996294 2603 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 01:50:35.996551 2603 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "9f86d7b4e82e43079d9a7d728aca3860"
format_stamp: "Formatted at 2026-09-29 01:50:26 on dist-test-slave-vz23"
I20260929 01:50:35.997407 2603 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:36.018293 2603 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:36.019264 2603 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:36.019546 2603 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:36.020578 2603 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:36.022468 2603 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:36.022585 2603 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:36.022722 2603 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:36.022805 2603 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:50:36.023177 2603 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:50:36.023281 2603 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:50:36.023375 2603 fault_injection.cc:59] injecting fault for kudu (pid 2603): FLAGS_fault_before_start (process will exit)
I20260929 01:50:36.042256 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:35189
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--webserver_port=44003
--webserver_interface=127.0.71.65
--time_source=system_unsync with env {}
I20260929 01:50:36.156873 2417 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [term 3 FOLLOWER]: Refusing update from remote peer 96a59e909fb44ae1bc1b7ac327df09b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 01:50:36.157734 2604 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:50:36.159647 1829 raft_consensus.cc:1275] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [term 3 FOLLOWER]: Refusing update from remote peer 96a59e909fb44ae1bc1b7ac327df09b5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 01:50:36.160411 2604 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:50:36.161469 2634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96a59e909fb44ae1bc1b7ac327df09b5. Latest consensus state: current_term: 3 leader_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:36.161800 2634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:36.163326 2606 sys_catalog.cc:455] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [sys.catalog]: SysCatalogTable state changed. Reason: New leader 96a59e909fb44ae1bc1b7ac327df09b5. Latest consensus state: current_term: 3 leader_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:36.163550 2606 sys_catalog.cc:458] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:36.163522 2609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:36.163836 2609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:36.164412 2604 sys_catalog.cc:455] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:36.164696 2604 sys_catalog.cc:458] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:36.165046 2634 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:36.165285 2634 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0ea38b4fa64841e8a36df8bb12a20110 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:36.166635 2606 sys_catalog.cc:455] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0ea38b4fa64841e8a36df8bb12a20110" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42419 } } peers { permanent_uuid: "a067d22fc04447ed8804d60c167c0bfc" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 43325 } } peers { permanent_uuid: "96a59e909fb44ae1bc1b7ac327df09b5" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40033 } } }
I20260929 01:50:36.166849 2606 sys_catalog.cc:458] T 00000000000000000000000000000000 P a067d22fc04447ed8804d60c167c0bfc [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:50:36.373574 2633 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:36.373930 2633 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:36.374065 2633 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:36.382730 2633 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:36.382962 2633 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:50:36.393101 2633 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:35189
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=44003
--tserver_master_addrs=127.0.71.126:42419,127.0.71.125:43325,127.0.71.124:40033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:36.394776 2633 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:36.396713 2633 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:36.406361 2646 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:36.406371 2645 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:36.407099 2648 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:36.407565 2633 server_base.cc:1061] running on GCE node
I20260929 01:50:36.408095 2633 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:50:36.408280 2633 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:50:36.408365 2633 hybrid_clock.cc:648] HybridClock initialized: now 1790646636408346 us; error 0 us; skew 500 ppm
I20260929 01:50:36.408718 2633 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:36.410892 2633 webserver.cc:533] Webserver started at http://127.0.71.65:44003/ using document root <none> and password file <none>
I20260929 01:50:36.411618 2633 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:36.411705 2633 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:36.417120 2633 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:50:36.420261 2653 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:36.422030 2633 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:50:36.422220 2633 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "9f86d7b4e82e43079d9a7d728aca3860"
format_stamp: "Formatted at 2026-09-29 01:50:26 on dist-test-slave-vz23"
I20260929 01:50:36.422976 2633 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:36.445401 2633 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:36.446318 2633 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:36.446614 2633 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:36.447649 2633 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:36.449486 2633 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:36.449558 2633 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:36.449708 2633 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:36.449759 2633 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:36.485136 2633 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:35189
I20260929 01:50:36.485175 2765 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:35189 every 8 connection(s)
I20260929 01:50:36.486621 2633 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:50:36.490479 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 2633
I20260929 01:50:36.490935 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 2633
I20260929 01:50:36.524701 2761 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:36.614500 2766 heartbeater.cc:344] Connected to a master server at 127.0.71.124:40033
I20260929 01:50:36.615195 2766 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:36.615731 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 2633
I20260929 01:50:36.635159 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 2097
W20260929 01:50:36.635763 2776 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:50:36.518541 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:43299 (local address 127.0.71.125:43325)
0929 01:50:36.519061 (+ 520us) server_negotiation.cc:207] Beginning negotiation
0929 01:50:36.519068 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0929 01:50:36.519836 (+ 768us) server_negotiation.cc:408] Connection header received
0929 01:50:36.520029 (+ 193us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:50:36.520035 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:50:36.520115 (+ 80us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:50:36.520220 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:50:36.521910 (+ 1690us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:50:36.523182 (+ 1272us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:50:36.525819 (+ 2637us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:50:36.526424 (+ 605us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0929 01:50:36.634851 (+108427us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:43299: BlockingRecv error: recv got EOF from 127.0.71.65:43299 (error 108)
Metrics: {"server-negotiator.queue_time_us":409,"thread_start_us":268,"threads_started":1}
W20260929 01:50:36.635800 2771 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:50:36.489925 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:46221 (local address 127.0.71.126:42419)
0929 01:50:36.503367 (+ 13442us) server_negotiation.cc:207] Beginning negotiation
0929 01:50:36.503387 (+ 20us) server_negotiation.cc:400] Waiting for connection header
0929 01:50:36.519881 (+ 16494us) server_negotiation.cc:408] Connection header received
0929 01:50:36.520142 (+ 261us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:50:36.520149 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:50:36.520238 (+ 89us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:50:36.520343 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:50:36.522595 (+ 2252us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:50:36.524024 (+ 1429us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:50:36.634835 (+110811us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:46221: BlockingRecv error: recv got EOF from 127.0.71.65:46221 (error 108)
Metrics: {"server-negotiator.queue_time_us":13287,"thread_start_us":20849,"threads_started":1}
I20260929 01:50:36.644050 2230 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:36.737202 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 2097
I20260929 01:50:36.753116 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 2241
I20260929 01:50:36.761173 2363 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:36.847201 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 2241
I20260929 01:50:36.863498 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 2376
I20260929 01:50:36.870278 2437 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:36.975029 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 2376
I20260929 01:50:36.997227 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 1773
I20260929 01:50:36.998338 1849 generic_service.cc:196] Checking for leaks (request via RPC)
W20260929 01:50:37.066457 1886 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.126:42419: connect: Connection refused (error 111)
W20260929 01:50:37.067854 1886 consensus_peers.cc:597] T 00000000000000000000000000000000 P 96a59e909fb44ae1bc1b7ac327df09b5 -> Peer 0ea38b4fa64841e8a36df8bb12a20110 (127.0.71.126:42419): Couldn't send request to peer 0ea38b4fa64841e8a36df8bb12a20110. Status: Network error: Client connection negotiation failed: client connection to 127.0.71.126:42419: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 01:50:37.110980 1856 master_runner.cc:187] Error getting master registration for 127.0.71.126:42419: Network error: Client connection negotiation failed: client connection to 127.0.71.126:42419: connect: Connection refused (error 111),
I20260929 01:50:37.114456 1856 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:50:37.117336 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 1773
I20260929 01:50:37.137029 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 1856
I20260929 01:50:37.138720 1932 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:37.245739 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 1856
[ OK ] ExternalMiniClusterTest.TestBasicOperation/0 (12932 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/1
2026-09-29T01:50:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:50:37Z Disabled control of system clock
I20260929 01:50:37.301755 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:40159
--webserver_interface=127.0.71.126
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879 with env {}
W20260929 01:50:37.625319 2791 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:37.625646 2791 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:37.625711 2791 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:37.634150 2791 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:50:37.634249 2791 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:37.634305 2791 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:50:37.634344 2791 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:50:37.644414 2791 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:40159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:37.645953 2791 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:37.647738 2791 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:37.655586 2797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:37.655586 2796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:37.656474 2799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:37.657162 2791 server_base.cc:1061] running on GCE node
I20260929 01:50:37.657990 2791 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:37.659408 2791 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:37.660617 2791 hybrid_clock.cc:648] HybridClock initialized: now 1790646637660585 us; error 66 us; skew 500 ppm
I20260929 01:50:37.661059 2791 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:37.663357 2791 webserver.cc:533] Webserver started at http://127.0.71.126:38103/ using document root <none> and password file <none>
I20260929 01:50:37.664089 2791 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:37.664224 2791 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:37.664649 2791 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:50:37.667668 2791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "7595c0707a67425096f7498511d0cbed"
format_stamp: "Formatted at 2026-09-29 01:50:37 on dist-test-slave-vz23"
I20260929 01:50:37.668736 2791 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "7595c0707a67425096f7498511d0cbed"
format_stamp: "Formatted at 2026-09-29 01:50:37 on dist-test-slave-vz23"
I20260929 01:50:37.674185 2791 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 01:50:37.677676 2805 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:37.679147 2791 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:50:37.679337 2791 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "7595c0707a67425096f7498511d0cbed"
format_stamp: "Formatted at 2026-09-29 01:50:37 on dist-test-slave-vz23"
I20260929 01:50:37.679550 2791 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:37.694721 2791 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:37.695537 2791 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:37.695822 2791 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:37.715911 2791 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:40159
I20260929 01:50:37.715962 2856 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:40159 every 8 connection(s)
I20260929 01:50:37.717504 2791 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:50:37.718786 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 2791
I20260929 01:50:37.719007 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:50:37.719964 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:41041
--webserver_interface=127.0.71.125
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879 with env {}
I20260929 01:50:37.722421 2857 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:50:37.732430 2857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:50:37.750325 2857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:50:37.751784 2807 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.125:41041: connect: Connection refused (error 111)
W20260929 01:50:37.753458 2857 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:41041: Network error: Client connection negotiation failed: client connection to 127.0.71.125:41041: connect: Connection refused (error 111)
I20260929 01:50:37.802850 2857 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } attempt: 1
W20260929 01:50:37.805160 2857 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:41041: Network error: Client connection negotiation failed: client connection to 127.0.71.125:41041: connect: Connection refused (error 111)
I20260929 01:50:37.873463 2857 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } attempt: 2
W20260929 01:50:37.875636 2857 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:41041: Network error: Client connection negotiation failed: client connection to 127.0.71.125:41041: connect: Connection refused (error 111)
I20260929 01:50:37.966912 2857 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } attempt: 3
W20260929 01:50:37.969050 2857 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:41041: Network error: Client connection negotiation failed: client connection to 127.0.71.125:41041: connect: Connection refused (error 111)
W20260929 01:50:38.060796 2859 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:38.061134 2859 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:38.061292 2859 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:38.069756 2859 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:50:38.069851 2859 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:38.069906 2859 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:50:38.069947 2859 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:50:38.080098 2859 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:41041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:38.081743 2859 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:38.083559 2859 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:38.092072 2868 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:38.092036 2867 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:38.093394 2870 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:38.094059 2859 server_base.cc:1061] running on GCE node
I20260929 01:50:38.094796 2859 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:38.096213 2859 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:38.097446 2859 hybrid_clock.cc:648] HybridClock initialized: now 1790646638097370 us; error 65 us; skew 500 ppm
I20260929 01:50:38.097818 2859 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:38.100132 2859 webserver.cc:533] Webserver started at http://127.0.71.125:42105/ using document root <none> and password file <none>
I20260929 01:50:38.100903 2859 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:38.101017 2859 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:38.101382 2859 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:50:38.103695 2859 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/data/instance:
uuid: "bd079154587d4fe3be45bde9d465902c"
format_stamp: "Formatted at 2026-09-29 01:50:38 on dist-test-slave-vz23"
I20260929 01:50:38.104425 2859 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/wal/instance:
uuid: "bd079154587d4fe3be45bde9d465902c"
format_stamp: "Formatted at 2026-09-29 01:50:38 on dist-test-slave-vz23"
I20260929 01:50:38.109687 2859 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 01:50:38.112355 2857 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } attempt: 4
I20260929 01:50:38.113695 2876 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20260929 01:50:38.114534 2857 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:41041: Network error: Client connection negotiation failed: client connection to 127.0.71.125:41041: connect: Connection refused (error 111)
I20260929 01:50:38.115330 2859 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:50:38.115509 2859 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "bd079154587d4fe3be45bde9d465902c"
format_stamp: "Formatted at 2026-09-29 01:50:38 on dist-test-slave-vz23"
I20260929 01:50:38.115741 2859 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:38.138038 2859 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:38.138934 2859 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:38.139230 2859 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:38.159916 2859 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:41041
I20260929 01:50:38.160084 2927 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:41041 every 8 connection(s)
I20260929 01:50:38.161500 2859 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:50:38.165865 2928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:50:38.171566 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 2859
I20260929 01:50:38.171757 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:50:38.172247 2928 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:50:38.173280 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:39879
--webserver_interface=127.0.71.124
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879 with env {}
I20260929 01:50:38.187152 2928 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:50:38.196267 2928 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:50:38.197444 2878 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.124:39879: connect: Connection refused (error 111)
W20260929 01:50:38.199370 2928 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:39879: Network error: Client connection negotiation failed: client connection to 127.0.71.124:39879: connect: Connection refused (error 111)
I20260929 01:50:38.249261 2928 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } attempt: 1
W20260929 01:50:38.251693 2928 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:39879: Network error: Client connection negotiation failed: client connection to 127.0.71.124:39879: connect: Connection refused (error 111)
I20260929 01:50:38.319952 2928 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } attempt: 2
W20260929 01:50:38.322134 2928 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:39879: Network error: Client connection negotiation failed: client connection to 127.0.71.124:39879: connect: Connection refused (error 111)
I20260929 01:50:38.413375 2928 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } attempt: 3
I20260929 01:50:38.413815 2857 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } attempt: 5
W20260929 01:50:38.415839 2928 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:39879: Network error: Client connection negotiation failed: client connection to 127.0.71.124:39879: connect: Connection refused (error 111)
I20260929 01:50:38.420511 2857 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:50:38.422561 2857 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:39879: Network error: Client connection negotiation failed: client connection to 127.0.71.124:39879: connect: Connection refused (error 111)
I20260929 01:50:38.473793 2857 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } attempt: 1
W20260929 01:50:38.475885 2857 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:39879: Network error: Client connection negotiation failed: client connection to 127.0.71.124:39879: connect: Connection refused (error 111)
W20260929 01:50:38.511016 2931 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:38.511421 2931 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:38.511562 2931 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:38.520519 2931 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:50:38.520658 2931 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:38.520756 2931 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:50:38.520848 2931 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:50:38.530949 2931 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:39879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:38.532747 2931 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:38.534677 2931 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:38.543485 2941 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:38.543453 2939 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:38.543622 2938 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:38.544111 2857 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } attempt: 2
I20260929 01:50:38.544745 2931 server_base.cc:1061] running on GCE node
I20260929 01:50:38.545776 2931 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20260929 01:50:38.546085 2857 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:39879: Network error: Client connection negotiation failed: client connection to 127.0.71.124:39879: connect: Connection refused (error 111)
I20260929 01:50:38.547382 2931 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:38.548597 2931 hybrid_clock.cc:648] HybridClock initialized: now 1790646638548541 us; error 63 us; skew 500 ppm
I20260929 01:50:38.548993 2931 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:38.551198 2931 webserver.cc:533] Webserver started at http://127.0.71.124:44857/ using document root <none> and password file <none>
I20260929 01:50:38.551864 2931 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:38.552078 2931 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:38.552397 2931 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:50:38.554723 2931 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/data/instance:
uuid: "39a6fd98553d489881677bfc15bc95ff"
format_stamp: "Formatted at 2026-09-29 01:50:38 on dist-test-slave-vz23"
I20260929 01:50:38.555444 2931 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/wal/instance:
uuid: "39a6fd98553d489881677bfc15bc95ff"
format_stamp: "Formatted at 2026-09-29 01:50:38 on dist-test-slave-vz23"
I20260929 01:50:38.559104 2928 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } attempt: 4
W20260929 01:50:38.561159 2928 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:39879: Network error: Client connection negotiation failed: client connection to 127.0.71.124:39879: connect: Connection refused (error 111)
I20260929 01:50:38.561235 2931 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 01:50:38.565212 2947 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:38.566687 2931 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:50:38.566892 2931 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "39a6fd98553d489881677bfc15bc95ff"
format_stamp: "Formatted at 2026-09-29 01:50:38 on dist-test-slave-vz23"
I20260929 01:50:38.567106 2931 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:38.581298 2931 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:38.582156 2931 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:38.582448 2931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:38.603636 2931 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:39879
I20260929 01:50:38.603655 2998 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:39879 every 8 connection(s)
I20260929 01:50:38.605454 2931 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:50:38.610076 2999 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:50:38.613513 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 2931
I20260929 01:50:38.613737 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:50:38.618093 2999 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:50:38.634114 2999 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:50:38.641285 2999 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:50:38.651961 2999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff: Bootstrap starting.
I20260929 01:50:38.652302 2857 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } attempt: 3
I20260929 01:50:38.655997 2999 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff: Neither blocks nor log segments found. Creating new log.
I20260929 01:50:38.657419 2999 log.cc:826] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff: Log is configured to *not* fsync() on all Append() calls
I20260929 01:50:38.660523 2999 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff: No bootstrap required, opened a new log
I20260929 01:50:38.663172 2857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: Bootstrap starting.
I20260929 01:50:38.666072 2999 raft_consensus.cc:359] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } }
I20260929 01:50:38.666436 2999 raft_consensus.cc:385] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:50:38.666574 2999 raft_consensus.cc:740] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39a6fd98553d489881677bfc15bc95ff, State: Initialized, Role: FOLLOWER
I20260929 01:50:38.666571 2857 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: Neither blocks nor log segments found. Creating new log.
I20260929 01:50:38.667207 2999 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } }
I20260929 01:50:38.667742 2857 log.cc:826] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: Log is configured to *not* fsync() on all Append() calls
I20260929 01:50:38.668166 3006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:38.668565 3006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:38.669430 2999 sys_catalog.cc:565] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:50:38.670658 2857 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: No bootstrap required, opened a new log
I20260929 01:50:38.677508 2857 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } }
I20260929 01:50:38.677958 2857 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:50:38.678143 2857 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7595c0707a67425096f7498511d0cbed, State: Initialized, Role: FOLLOWER
I20260929 01:50:38.678920 2857 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } }
I20260929 01:50:38.679895 3019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:38.680276 3019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:38.681563 2857 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:50:38.678642 2931 master_runner.cc:187] Error getting master registration for 127.0.71.126:40159: OK, instance_id { permanent_uuid: "7595c0707a67425096f7498511d0cbed" instance_seqno: 1790646637710326 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:50:38.683511 2931 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:50:38.688735 3018 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:50:38.688860 3018 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 01:50:38.698383 2811 tablet.cc:2433] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:50:38.704334 3031 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:50:38.704531 3031 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:50:38.703087 2791 master_runner.cc:187] Error getting master registration for 127.0.71.125:41041: OK, instance_id { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" instance_seqno: 1790646638154112 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:50:38.709538 2791 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:50:38.834604 2953 tablet.cc:2433] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:50:38.860405 2928 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } attempt: 5
I20260929 01:50:38.874356 2928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c: Bootstrap starting.
I20260929 01:50:38.878747 2928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c: Neither blocks nor log segments found. Creating new log.
I20260929 01:50:38.880105 2928 log.cc:826] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c: Log is configured to *not* fsync() on all Append() calls
I20260929 01:50:38.883289 2928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c: No bootstrap required, opened a new log
I20260929 01:50:38.891232 2928 raft_consensus.cc:359] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } }
I20260929 01:50:38.891680 2928 raft_consensus.cc:385] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:50:38.891810 2928 raft_consensus.cc:740] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd079154587d4fe3be45bde9d465902c, State: Initialized, Role: FOLLOWER
I20260929 01:50:38.892575 2928 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } }
I20260929 01:50:38.893607 3034 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:38.893956 3034 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:38.894886 2928 sys_catalog.cc:565] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:50:38.912635 2859 master_runner.cc:264] No leader master found from master bd079154587d4fe3be45bde9d465902c
I20260929 01:50:38.912770 2859 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:50:38.915556 3045 catalog_manager.cc:1607] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:50:38.915728 3045 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:50:39.009864 3034 raft_consensus.cc:493] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:50:39.010141 3034 raft_consensus.cc:515] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } }
I20260929 01:50:39.011879 3034 leader_election.cc:290] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7595c0707a67425096f7498511d0cbed (127.0.71.126:40159), 39a6fd98553d489881677bfc15bc95ff (127.0.71.124:39879)
I20260929 01:50:39.014477 2974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd079154587d4fe3be45bde9d465902c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39a6fd98553d489881677bfc15bc95ff" is_pre_election: true
I20260929 01:50:39.015009 2974 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd079154587d4fe3be45bde9d465902c in term 0.
I20260929 01:50:39.015990 2832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd079154587d4fe3be45bde9d465902c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7595c0707a67425096f7498511d0cbed" is_pre_election: true
I20260929 01:50:39.016515 2832 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd079154587d4fe3be45bde9d465902c in term 0.
I20260929 01:50:39.017441 2879 leader_election.cc:304] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7595c0707a67425096f7498511d0cbed, bd079154587d4fe3be45bde9d465902c; no voters:
I20260929 01:50:39.019739 3034 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:50:39.019891 3034 raft_consensus.cc:493] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:50:39.019989 3034 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:50:39.023460 3034 raft_consensus.cc:515] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } }
I20260929 01:50:39.024232 3034 leader_election.cc:290] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [CANDIDATE]: Term 1 election: Requested vote from peers 7595c0707a67425096f7498511d0cbed (127.0.71.126:40159), 39a6fd98553d489881677bfc15bc95ff (127.0.71.124:39879)
I20260929 01:50:39.025928 2832 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd079154587d4fe3be45bde9d465902c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7595c0707a67425096f7498511d0cbed"
I20260929 01:50:39.026152 2832 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:50:39.028738 2974 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd079154587d4fe3be45bde9d465902c" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "39a6fd98553d489881677bfc15bc95ff"
I20260929 01:50:39.029006 2974 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:50:39.034049 2832 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd079154587d4fe3be45bde9d465902c in term 1.
I20260929 01:50:39.037288 2879 leader_election.cc:304] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7595c0707a67425096f7498511d0cbed, bd079154587d4fe3be45bde9d465902c; no voters:
I20260929 01:50:39.037863 3034 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:50:39.038295 2974 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd079154587d4fe3be45bde9d465902c in term 1.
I20260929 01:50:39.044735 3034 raft_consensus.cc:697] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [term 1 LEADER]: Becoming Leader. State: Replica: bd079154587d4fe3be45bde9d465902c, State: Running, Role: LEADER
I20260929 01:50:39.045378 3034 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } }
I20260929 01:50:39.062044 3047 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd079154587d4fe3be45bde9d465902c. Latest consensus state: current_term: 1 leader_uuid: "bd079154587d4fe3be45bde9d465902c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:39.062405 3047 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:39.063333 3049 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:50:39.066097 3049 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:50:39.081758 2832 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [term 1 FOLLOWER]: Refusing update from remote peer bd079154587d4fe3be45bde9d465902c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:50:39.081751 2974 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [term 1 FOLLOWER]: Refusing update from remote peer bd079154587d4fe3be45bde9d465902c: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:50:39.082808 3047 consensus_queue.cc:1048] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [LEADER]: Connected to new peer: Peer: permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:50:39.083539 3034 consensus_queue.cc:1048] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:50:39.108146 3006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd079154587d4fe3be45bde9d465902c. Latest consensus state: current_term: 1 leader_uuid: "bd079154587d4fe3be45bde9d465902c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:39.109119 3006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:39.120133 3050 mvcc.cc:204] Tried to move back new op lower bound from 7334488633666039808 to 7334488633536507904. Current Snapshot: MvccSnapshot[applied={T|T < 7334488633666039808}]
I20260929 01:50:39.128995 3006 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd079154587d4fe3be45bde9d465902c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:39.129221 3006 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:39.137660 3034 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd079154587d4fe3be45bde9d465902c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:39.137989 3034 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:39.138000 3019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd079154587d4fe3be45bde9d465902c. Latest consensus state: current_term: 1 leader_uuid: "bd079154587d4fe3be45bde9d465902c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:39.138214 3019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:39.140280 3049 catalog_manager.cc:1396] Generated new cluster ID: 59443e495098424cb36cef5e64d4d4bb
I20260929 01:50:39.140399 3049 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:50:39.149665 3057 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "bd079154587d4fe3be45bde9d465902c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:39.149998 3057 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:39.169170 3019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "bd079154587d4fe3be45bde9d465902c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:39.169425 3019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:39.178570 3049 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 01:50:39.180238 3049 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:50:39.212327 3049 catalog_manager.cc:6219] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c: Generated new TSK 0
I20260929 01:50:39.213594 3049 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:50:39.234567 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--builtin_ntp_servers=127.0.71.84:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260929 01:50:39.579242 3059 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:39.579587 3059 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:39.579731 3059 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:39.588403 3059 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:39.588733 3059 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:50:39.598855 3059 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:39.600584 3059 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:39.602504 3059 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:39.612116 3065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:39.612811 3064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:39.617983 3067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:39.618434 3059 server_base.cc:1061] running on GCE node
I20260929 01:50:39.619140 3059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:39.620319 3059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:39.621543 3059 hybrid_clock.cc:648] HybridClock initialized: now 1790646639621502 us; error 78 us; skew 500 ppm
I20260929 01:50:39.621953 3059 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:39.624548 3059 webserver.cc:533] Webserver started at http://127.0.71.65:37255/ using document root <none> and password file <none>
I20260929 01:50:39.625422 3059 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:39.625528 3059 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:39.625909 3059 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:50:39.629012 3059 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "4407206f6d574cfb95ac7d10f259df44"
format_stamp: "Formatted at 2026-09-29 01:50:39 on dist-test-slave-vz23"
I20260929 01:50:39.629940 3059 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "4407206f6d574cfb95ac7d10f259df44"
format_stamp: "Formatted at 2026-09-29 01:50:39 on dist-test-slave-vz23"
I20260929 01:50:39.635637 3059 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:50:39.639062 3073 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:39.640545 3059 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:50:39.640736 3059 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "4407206f6d574cfb95ac7d10f259df44"
format_stamp: "Formatted at 2026-09-29 01:50:39 on dist-test-slave-vz23"
I20260929 01:50:39.641012 3059 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:39.663169 3059 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:39.664081 3059 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:39.664367 3059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:39.665367 3059 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:39.667039 3059 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:39.667161 3059 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:39.667286 3059 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:39.667373 3059 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:39.691505 3018 catalog_manager.cc:1308] Loaded cluster ID: 59443e495098424cb36cef5e64d4d4bb
I20260929 01:50:39.691661 3018 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff: loading cluster ID for follower catalog manager: success
I20260929 01:50:39.695847 3018 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff: acquiring CA information for follower catalog manager: success
I20260929 01:50:39.698653 3018 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:50:39.706967 3031 catalog_manager.cc:1308] Loaded cluster ID: 59443e495098424cb36cef5e64d4d4bb
I20260929 01:50:39.707086 3031 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: loading cluster ID for follower catalog manager: success
I20260929 01:50:39.709052 3059 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:36117
I20260929 01:50:39.709110 3185 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:36117 every 8 connection(s)
I20260929 01:50:39.710584 3059 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:50:39.710709 3031 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: acquiring CA information for follower catalog manager: success
I20260929 01:50:39.713523 3031 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:50:39.716708 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 3059
I20260929 01:50:39.716902 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
I20260929 01:50:39.723668 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.66:0
--local_ip_for_outbound_sockets=127.0.71.66
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--builtin_ntp_servers=127.0.71.84:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:50:39.740924 3187 heartbeater.cc:344] Connected to a master server at 127.0.71.126:40159
I20260929 01:50:39.741283 3187 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:39.742398 3187 heartbeater.cc:507] Master 127.0.71.126:40159 requested a full tablet report, sending...
I20260929 01:50:39.745497 2822 ts_manager.cc:194] Registered new tserver with Master: 4407206f6d574cfb95ac7d10f259df44 (127.0.71.65:36117)
I20260929 01:50:39.748584 3186 heartbeater.cc:344] Connected to a master server at 127.0.71.124:39879
I20260929 01:50:39.748735 3186 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:39.749292 3186 heartbeater.cc:507] Master 127.0.71.124:39879 requested a full tablet report, sending...
I20260929 01:50:39.751221 2964 ts_manager.cc:194] Registered new tserver with Master: 4407206f6d574cfb95ac7d10f259df44 (127.0.71.65:36117)
I20260929 01:50:39.755508 3188 heartbeater.cc:344] Connected to a master server at 127.0.71.125:41041
I20260929 01:50:39.755656 3188 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:39.756100 3188 heartbeater.cc:507] Master 127.0.71.125:41041 requested a full tablet report, sending...
I20260929 01:50:39.757478 2893 ts_manager.cc:194] Registered new tserver with Master: 4407206f6d574cfb95ac7d10f259df44 (127.0.71.65:36117)
I20260929 01:50:39.758929 2893 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:37657
W20260929 01:50:40.063179 3196 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:40.063539 3196 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:40.063674 3196 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:40.072204 3196 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:40.072476 3196 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.66
I20260929 01:50:40.082909 3196 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:40.084604 3196 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:40.086463 3196 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:40.096932 3204 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:40.096675 3202 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:40.096712 3201 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:40.098189 3196 server_base.cc:1061] running on GCE node
I20260929 01:50:40.098760 3196 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:40.099781 3196 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:40.101018 3196 hybrid_clock.cc:648] HybridClock initialized: now 1790646640100986 us; error 48 us; skew 500 ppm
I20260929 01:50:40.101361 3196 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:40.103839 3196 webserver.cc:533] Webserver started at http://127.0.71.66:39167/ using document root <none> and password file <none>
I20260929 01:50:40.104628 3196 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:40.104761 3196 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:40.105122 3196 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:50:40.107481 3196 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/data/instance:
uuid: "0b15cef5b60b4e8db93620cd7d94d200"
format_stamp: "Formatted at 2026-09-29 01:50:40 on dist-test-slave-vz23"
I20260929 01:50:40.108259 3196 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/wal/instance:
uuid: "0b15cef5b60b4e8db93620cd7d94d200"
format_stamp: "Formatted at 2026-09-29 01:50:40 on dist-test-slave-vz23"
I20260929 01:50:40.113512 3196 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 01:50:40.117053 3210 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:40.118491 3196 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 01:50:40.118685 3196 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/wal
uuid: "0b15cef5b60b4e8db93620cd7d94d200"
format_stamp: "Formatted at 2026-09-29 01:50:40 on dist-test-slave-vz23"
I20260929 01:50:40.118891 3196 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:40.145855 3196 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:40.146982 3196 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:40.147274 3196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:40.148481 3196 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:40.150352 3196 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:40.150476 3196 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:40.150588 3196 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:40.150661 3196 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:40.188262 3196 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.66:34011
I20260929 01:50:40.188354 3322 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.66:34011 every 8 connection(s)
I20260929 01:50:40.189780 3196 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
I20260929 01:50:40.191241 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 3196
I20260929 01:50:40.191469 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/wal/instance
I20260929 01:50:40.196673 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.67:0
--local_ip_for_outbound_sockets=127.0.71.67
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--builtin_ntp_servers=127.0.71.84:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:50:40.212041 3323 heartbeater.cc:344] Connected to a master server at 127.0.71.124:39879
I20260929 01:50:40.212358 3324 heartbeater.cc:344] Connected to a master server at 127.0.71.126:40159
I20260929 01:50:40.212482 3323 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:40.212606 3324 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:40.213591 3325 heartbeater.cc:344] Connected to a master server at 127.0.71.125:41041
I20260929 01:50:40.213727 3325 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:40.213775 3324 heartbeater.cc:507] Master 127.0.71.126:40159 requested a full tablet report, sending...
I20260929 01:50:40.213819 3323 heartbeater.cc:507] Master 127.0.71.124:39879 requested a full tablet report, sending...
I20260929 01:50:40.214286 3325 heartbeater.cc:507] Master 127.0.71.125:41041 requested a full tablet report, sending...
I20260929 01:50:40.215581 2893 ts_manager.cc:194] Registered new tserver with Master: 0b15cef5b60b4e8db93620cd7d94d200 (127.0.71.66:34011)
I20260929 01:50:40.216049 2822 ts_manager.cc:194] Registered new tserver with Master: 0b15cef5b60b4e8db93620cd7d94d200 (127.0.71.66:34011)
I20260929 01:50:40.216511 2893 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.66:43647
I20260929 01:50:40.217687 2964 ts_manager.cc:194] Registered new tserver with Master: 0b15cef5b60b4e8db93620cd7d94d200 (127.0.71.66:34011)
W20260929 01:50:40.540112 3330 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:40.540481 3330 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:40.540622 3330 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:40.548998 3330 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:40.549211 3330 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.67
I20260929 01:50:40.559085 3330 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:40.560778 3330 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:40.562583 3330 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:40.571945 3335 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:40.572031 3336 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:40.572616 3338 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:40.573076 3330 server_base.cc:1061] running on GCE node
I20260929 01:50:40.573657 3330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:40.574676 3330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:40.575954 3330 hybrid_clock.cc:648] HybridClock initialized: now 1790646640575912 us; error 72 us; skew 500 ppm
I20260929 01:50:40.576275 3330 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:40.578578 3330 webserver.cc:533] Webserver started at http://127.0.71.67:38055/ using document root <none> and password file <none>
I20260929 01:50:40.579473 3330 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:40.579608 3330 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:40.580003 3330 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:50:40.582638 3330 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/data/instance:
uuid: "e5477c1a84e244019033fc3ea62d98b6"
format_stamp: "Formatted at 2026-09-29 01:50:40 on dist-test-slave-vz23"
I20260929 01:50:40.583539 3330 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/wal/instance:
uuid: "e5477c1a84e244019033fc3ea62d98b6"
format_stamp: "Formatted at 2026-09-29 01:50:40 on dist-test-slave-vz23"
I20260929 01:50:40.589322 3330 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 01:50:40.592785 3344 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:40.594270 3330 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:50:40.594453 3330 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/wal
uuid: "e5477c1a84e244019033fc3ea62d98b6"
format_stamp: "Formatted at 2026-09-29 01:50:40 on dist-test-slave-vz23"
I20260929 01:50:40.594668 3330 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:40.624104 3330 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:40.625118 3330 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:40.625403 3330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:40.626435 3330 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:40.628069 3330 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:40.628213 3330 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:40.628361 3330 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:40.628472 3330 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:40.663157 3330 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.67:42825
I20260929 01:50:40.663244 3456 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.67:42825 every 8 connection(s)
I20260929 01:50:40.664724 3330 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
I20260929 01:50:40.674045 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 3330
I20260929 01:50:40.674234 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/wal/instance
I20260929 01:50:40.684319 3459 heartbeater.cc:344] Connected to a master server at 127.0.71.125:41041
I20260929 01:50:40.684551 3457 heartbeater.cc:344] Connected to a master server at 127.0.71.124:39879
I20260929 01:50:40.684782 3457 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:40.684784 3459 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:40.685286 3458 heartbeater.cc:344] Connected to a master server at 127.0.71.126:40159
I20260929 01:50:40.685452 3458 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:40.685917 3459 heartbeater.cc:507] Master 127.0.71.125:41041 requested a full tablet report, sending...
I20260929 01:50:40.686043 3458 heartbeater.cc:507] Master 127.0.71.126:40159 requested a full tablet report, sending...
I20260929 01:50:40.687178 3457 heartbeater.cc:507] Master 127.0.71.124:39879 requested a full tablet report, sending...
I20260929 01:50:40.687425 2893 ts_manager.cc:194] Registered new tserver with Master: e5477c1a84e244019033fc3ea62d98b6 (127.0.71.67:42825)
I20260929 01:50:40.687618 2822 ts_manager.cc:194] Registered new tserver with Master: e5477c1a84e244019033fc3ea62d98b6 (127.0.71.67:42825)
I20260929 01:50:40.688215 2964 ts_manager.cc:194] Registered new tserver with Master: e5477c1a84e244019033fc3ea62d98b6 (127.0.71.67:42825)
I20260929 01:50:40.688378 2893 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.67:37369
I20260929 01:50:40.698927 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:50:40.705834 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:50:40.706029 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 2791
I20260929 01:50:40.706885 2852 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:40.761521 3188 heartbeater.cc:499] Master 127.0.71.125:41041 was elected leader, sending a full tablet report...
I20260929 01:50:40.819213 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 2791
I20260929 01:50:40.840166 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:40159
--webserver_interface=127.0.71.126
--webserver_port=38103
--builtin_ntp_servers=127.0.71.84:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879 with env {}
W20260929 01:50:41.181861 3465 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:41.182173 3465 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:41.182363 3465 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:41.191123 3465 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:50:41.191252 3465 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:41.191395 3465 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:50:41.191481 3465 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:50:41.202144 3465 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:40159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=38103
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:41.203823 3465 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:41.205803 3465 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:41.214890 3470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:41.214928 3471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:41.219081 3325 heartbeater.cc:499] Master 127.0.71.125:41041 was elected leader, sending a full tablet report...
W20260929 01:50:41.219835 3324 heartbeater.cc:646] Failed to heartbeat to 127.0.71.126:40159 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.71.126:40159: connect: Connection refused (error 111)
W20260929 01:50:41.220422 3473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:41.221171 3465 server_base.cc:1061] running on GCE node
I20260929 01:50:41.221984 3465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:41.223456 3465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:41.224720 3465 hybrid_clock.cc:648] HybridClock initialized: now 1790646641224650 us; error 56 us; skew 500 ppm
I20260929 01:50:41.225203 3465 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:41.227516 3465 webserver.cc:533] Webserver started at http://127.0.71.126:38103/ using document root <none> and password file <none>
I20260929 01:50:41.228536 3465 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:41.228691 3465 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:41.234633 3465 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:50:41.237813 3480 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:41.239383 3465 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:50:41.239578 3465 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "7595c0707a67425096f7498511d0cbed"
format_stamp: "Formatted at 2026-09-29 01:50:37 on dist-test-slave-vz23"
I20260929 01:50:41.240330 3465 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:41.259721 3465 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:41.260702 3465 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:41.261022 3465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:41.282023 3465 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:40159
I20260929 01:50:41.282066 3531 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:40159 every 8 connection(s)
I20260929 01:50:41.283746 3465 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:50:41.289775 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 3465
I20260929 01:50:41.290159 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 3059
I20260929 01:50:41.291381 3532 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:50:41.294900 3532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: Bootstrap starting.
I20260929 01:50:41.299412 3176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:41.314592 3532 log.cc:826] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: Log is configured to *not* fsync() on all Append() calls
I20260929 01:50:41.323729 3532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:50:41.324524 3532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: Bootstrap complete.
I20260929 01:50:41.331826 3532 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } }
I20260929 01:50:41.332653 3532 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7595c0707a67425096f7498511d0cbed, State: Initialized, Role: FOLLOWER
I20260929 01:50:41.333535 3532 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } }
I20260929 01:50:41.334437 3538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:41.334740 3538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:41.335491 3532 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:50:41.347453 3549 catalog_manager.cc:1308] Loaded cluster ID: 59443e495098424cb36cef5e64d4d4bb
I20260929 01:50:41.347698 3549 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: loading cluster ID for follower catalog manager: success
I20260929 01:50:41.353600 3549 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: acquiring CA information for follower catalog manager: success
I20260929 01:50:41.357312 3549 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:50:41.392545 3538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd079154587d4fe3be45bde9d465902c. Latest consensus state: current_term: 1 leader_uuid: "bd079154587d4fe3be45bde9d465902c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:41.392818 3538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:41.422523 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 3059
I20260929 01:50:41.441560 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:36117
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--webserver_port=37255
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:50:41.690311 3459 heartbeater.cc:499] Master 127.0.71.125:41041 was elected leader, sending a full tablet report...
I20260929 01:50:41.702803 3497 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e5477c1a84e244019033fc3ea62d98b6" instance_seqno: 1790646640654850) as {username='slave'} at 127.0.71.67:55381; Asking this server to re-register.
I20260929 01:50:41.703658 3458 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:41.705464 3497 ts_manager.cc:194] Registered new tserver with Master: e5477c1a84e244019033fc3ea62d98b6 (127.0.71.67:42825)
W20260929 01:50:41.812245 3557 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:41.812546 3557 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:41.812618 3557 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:41.820914 3557 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:41.821089 3557 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:50:41.831182 3557 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:36117
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=37255
--tserver_master_addrs=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:41.832695 3557 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:41.834396 3557 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:41.844787 3564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:41.844750 3563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:41.845685 3566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:41.846467 3557 server_base.cc:1061] running on GCE node
I20260929 01:50:41.847064 3557 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:41.848014 3557 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:41.849246 3557 hybrid_clock.cc:648] HybridClock initialized: now 1790646641849196 us; error 34 us; skew 500 ppm
I20260929 01:50:41.849622 3557 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:41.852097 3557 webserver.cc:533] Webserver started at http://127.0.71.65:37255/ using document root <none> and password file <none>
I20260929 01:50:41.852941 3557 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:41.853113 3557 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:41.858672 3557 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260929 01:50:41.861871 3572 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:41.863543 3557 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 01:50:41.863746 3557 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "4407206f6d574cfb95ac7d10f259df44"
format_stamp: "Formatted at 2026-09-29 01:50:39 on dist-test-slave-vz23"
I20260929 01:50:41.864542 3557 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:41.903406 3557 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:41.904289 3557 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:41.904640 3557 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:41.905604 3557 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:41.907253 3557 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:41.907363 3557 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:41.907503 3557 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:41.907573 3557 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:41.944828 3557 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:36117
I20260929 01:50:41.944957 3684 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:36117 every 8 connection(s)
I20260929 01:50:41.946393 3557 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:50:41.953667 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 3557
I20260929 01:50:41.954051 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 3557
I20260929 01:50:41.966656 3687 heartbeater.cc:344] Connected to a master server at 127.0.71.125:41041
I20260929 01:50:41.966897 3680 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:41.967124 3687 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:42.053444 3686 heartbeater.cc:344] Connected to a master server at 127.0.71.126:40159
I20260929 01:50:42.054909 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 3557
I20260929 01:50:42.055051 3687 heartbeater.cc:507] Master 127.0.71.125:41041 requested a full tablet report, sending...
I20260929 01:50:42.068257 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:36117
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--webserver_port=37255
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {}
W20260929 01:50:42.068672 3693 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:50:41.953101 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:51547 (local address 127.0.71.124:39879)
0929 01:50:41.963743 (+ 10642us) server_negotiation.cc:207] Beginning negotiation
0929 01:50:41.963750 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0929 01:50:41.963774 (+ 24us) server_negotiation.cc:408] Connection header received
0929 01:50:41.963838 (+ 64us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:50:41.963847 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:50:41.963925 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:50:41.966041 (+ 2116us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:50:41.967071 (+ 1030us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:50:41.968773 (+ 1702us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:50:42.067635 (+ 98862us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:51547: BlockingRecv error: recv got EOF from 127.0.71.65:51547 (error 108)
Metrics: {"server-negotiator.queue_time_us":10499,"thread_start_us":10406,"threads_started":1}
I20260929 01:50:42.229506 3324 heartbeater.cc:344] Connected to a master server at 127.0.71.126:40159
I20260929 01:50:42.230854 3497 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "0b15cef5b60b4e8db93620cd7d94d200" instance_seqno: 1790646640179347) as {username='slave'} at 127.0.71.66:43719; Asking this server to re-register.
I20260929 01:50:42.231534 3324 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:42.232836 3497 ts_manager.cc:194] Registered new tserver with Master: 0b15cef5b60b4e8db93620cd7d94d200 (127.0.71.66:34011)
W20260929 01:50:42.403779 3697 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:42.404163 3697 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:42.404305 3697 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:50:42.404402 3697 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:42.413390 3697 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:42.413682 3697 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:50:42.424656 3697 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:36117
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=37255
--tserver_master_addrs=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:42.426430 3697 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:42.428287 3697 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:42.438052 3704 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:42.438822 3703 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:42.439153 3706 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:42.439702 3697 server_base.cc:1061] running on GCE node
I20260929 01:50:42.440372 3697 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:42.441491 3697 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:42.442849 3697 hybrid_clock.cc:648] HybridClock initialized: now 1790646642442708 us; error 45 us; skew 500 ppm
I20260929 01:50:42.443184 3697 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:42.445760 3697 webserver.cc:533] Webserver started at http://127.0.71.65:37255/ using document root <none> and password file <none>
I20260929 01:50:42.446542 3697 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:42.446691 3697 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:42.452497 3697 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:50:42.455535 3712 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:42.457093 3697 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:50:42.457288 3697 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "4407206f6d574cfb95ac7d10f259df44"
format_stamp: "Formatted at 2026-09-29 01:50:39 on dist-test-slave-vz23"
I20260929 01:50:42.458025 3697 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:42.476303 3697 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:42.477207 3697 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:42.477481 3697 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:42.478425 3697 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:42.480093 3697 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:42.480221 3697 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:42.480345 3697 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:42.480434 3697 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:50:42.480844 3697 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:50:42.481047 3697 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:50:42.481133 3697 fault_injection.cc:59] injecting fault for kudu (pid 3697): FLAGS_fault_before_start (process will exit)
I20260929 01:50:42.497284 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:36117
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--webserver_port=37255
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:39955
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260929 01:50:42.833360 3720 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:42.833745 3720 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:42.833887 3720 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:42.842605 3720 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:42.842856 3720 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:50:42.853228 3720 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:39955
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:36117
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=37255
--tserver_master_addrs=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:42.854928 3720 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:42.856753 3720 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:42.866392 3726 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:42.866556 3728 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:42.866345 3725 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:42.867712 3720 server_base.cc:1061] running on GCE node
I20260929 01:50:42.868347 3720 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:42.869334 3720 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:42.870601 3720 hybrid_clock.cc:648] HybridClock initialized: now 1790646642870512 us; error 79 us; skew 500 ppm
I20260929 01:50:42.871003 3720 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:42.873543 3720 webserver.cc:533] Webserver started at http://127.0.71.65:37255/ using document root <none> and password file <none>
I20260929 01:50:42.874387 3720 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:42.874531 3720 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:42.879979 3720 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 01:50:42.882807 3734 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:42.884416 3720 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:50:42.884704 3720 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "4407206f6d574cfb95ac7d10f259df44"
format_stamp: "Formatted at 2026-09-29 01:50:39 on dist-test-slave-vz23"
I20260929 01:50:42.885602 3720 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:42.899191 3720 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:42.900161 3720 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:42.900542 3720 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:42.901527 3720 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:42.903183 3720 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:42.903290 3720 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:42.903443 3720 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:42.903515 3720 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:42.939744 3720 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:36117
I20260929 01:50:42.939764 3846 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:36117 every 8 connection(s)
I20260929 01:50:42.941282 3720 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:50:42.947110 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 3720
I20260929 01:50:42.947613 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 3720
I20260929 01:50:42.970768 3848 heartbeater.cc:344] Connected to a master server at 127.0.71.126:40159
I20260929 01:50:42.971266 3848 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:42.972528 3848 heartbeater.cc:507] Master 127.0.71.126:40159 requested a full tablet report, sending...
I20260929 01:50:42.972782 3847 heartbeater.cc:344] Connected to a master server at 127.0.71.124:39879
I20260929 01:50:42.972944 3847 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:42.973616 3847 heartbeater.cc:507] Master 127.0.71.124:39879 requested a full tablet report, sending...
I20260929 01:50:42.974349 3842 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:42.974512 3497 ts_manager.cc:194] Registered new tserver with Master: 4407206f6d574cfb95ac7d10f259df44 (127.0.71.65:36117)
I20260929 01:50:42.975553 2964 ts_manager.cc:194] Re-registered known tserver with Master: 4407206f6d574cfb95ac7d10f259df44 (127.0.71.65:36117)
I20260929 01:50:43.068423 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 3720
I20260929 01:50:43.085621 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 3196
W20260929 01:50:43.086540 3857 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:50:42.958489 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:52297 (local address 127.0.71.125:41041)
0929 01:50:42.971929 (+ 13440us) server_negotiation.cc:207] Beginning negotiation
0929 01:50:42.971938 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0929 01:50:42.971959 (+ 21us) server_negotiation.cc:408] Connection header received
0929 01:50:42.972017 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:50:42.972023 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:50:42.972107 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:50:42.972232 (+ 125us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:50:42.973934 (+ 1702us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:50:42.975285 (+ 1351us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:50:43.068499 (+ 93214us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:50:43.069041 (+ 542us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0929 01:50:43.085500 (+ 16459us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:52297: BlockingRecv error: recv got EOF from 127.0.71.65:52297 (error 108)
Metrics: {"server-negotiator.queue_time_us":13302,"thread_start_us":13850,"threads_started":1}
I20260929 01:50:43.094195 3318 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:43.182679 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 3196
I20260929 01:50:43.199064 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 3330
I20260929 01:50:43.207317 3452 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:43.297535 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 3330
I20260929 01:50:43.314338 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 3465
I20260929 01:50:43.320626 3527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:43.423243 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 3465
I20260929 01:50:43.443086 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 2859
I20260929 01:50:43.444355 2923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:43.550747 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 2859
I20260929 01:50:43.570346 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 2931
I20260929 01:50:43.571619 2994 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:43.675609 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 2931
2026-09-29T01:50:43Z chronyd exiting
2026-09-29T01:50:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:50:43Z Disabled control of system clock
I20260929 01:50:43.730005 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:40159
--webserver_interface=127.0.71.126
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:33125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879 with env {}
W20260929 01:50:44.070636 3870 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:44.071043 3870 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:44.071180 3870 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:44.079550 3870 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:50:44.079702 3870 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:44.079802 3870 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:50:44.079895 3870 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:50:44.090009 3870 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:40159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:44.091683 3870 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:44.093494 3870 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:44.101511 3876 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:44.101549 3875 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:44.102455 3878 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:44.102880 3870 server_base.cc:1061] running on GCE node
I20260929 01:50:44.103654 3870 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:44.105069 3870 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:44.106312 3870 hybrid_clock.cc:648] HybridClock initialized: now 1790646644106229 us; error 36 us; skew 500 ppm
I20260929 01:50:44.106671 3870 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:44.108997 3870 webserver.cc:533] Webserver started at http://127.0.71.126:44505/ using document root <none> and password file <none>
I20260929 01:50:44.109737 3870 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:44.109836 3870 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:44.115629 3870 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:50:44.118671 3884 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:44.120127 3870 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:50:44.120299 3870 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "7595c0707a67425096f7498511d0cbed"
format_stamp: "Formatted at 2026-09-29 01:50:37 on dist-test-slave-vz23"
I20260929 01:50:44.121104 3870 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:44.138438 3870 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:44.139290 3870 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:44.139585 3870 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:44.159047 3870 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:40159
I20260929 01:50:44.159075 3935 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:40159 every 8 connection(s)
I20260929 01:50:44.160669 3870 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:50:44.166956 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 3870
I20260929 01:50:44.167174 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:50:44.167627 3936 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:50:44.168146 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:41041
--webserver_interface=127.0.71.125
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:33125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879 with env {}
I20260929 01:50:44.171142 3936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: Bootstrap starting.
I20260929 01:50:44.187330 3936 log.cc:826] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: Log is configured to *not* fsync() on all Append() calls
I20260929 01:50:44.195700 3936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:50:44.196228 3936 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: Bootstrap complete.
I20260929 01:50:44.202850 3936 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } }
I20260929 01:50:44.203522 3936 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7595c0707a67425096f7498511d0cbed, State: Initialized, Role: FOLLOWER
I20260929 01:50:44.204262 3936 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } }
I20260929 01:50:44.205134 3940 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:44.205440 3940 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:44.206214 3936 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:50:44.217501 3951 catalog_manager.cc:1308] Loaded cluster ID: 59443e495098424cb36cef5e64d4d4bb
I20260929 01:50:44.217618 3951 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: loading cluster ID for follower catalog manager: success
I20260929 01:50:44.221665 3951 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: acquiring CA information for follower catalog manager: success
I20260929 01:50:44.224244 3951 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:50:44.243801 3870 master_runner.cc:187] Error getting master registration for 127.0.71.125:41041: Network error: Client connection negotiation failed: client connection to 127.0.71.125:41041: connect: Connection refused (error 111),
I20260929 01:50:44.249699 3870 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:50:44.511493 3938 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:44.511804 3938 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:44.511941 3938 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:44.520604 3938 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:50:44.520695 3938 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:44.520774 3938 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:50:44.520866 3938 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:50:44.531904 3938 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:41041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:44.533706 3938 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:44.535537 3938 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:44.543339 3959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:44.545248 3962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:44.545543 3960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:44.545930 3938 server_base.cc:1061] running on GCE node
I20260929 01:50:44.546690 3938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:44.547976 3938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:44.549222 3938 hybrid_clock.cc:648] HybridClock initialized: now 1790646644549162 us; error 41 us; skew 500 ppm
I20260929 01:50:44.549636 3938 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:44.551930 3938 webserver.cc:533] Webserver started at http://127.0.71.125:35139/ using document root <none> and password file <none>
I20260929 01:50:44.552718 3938 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:44.552866 3938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:44.558437 3938 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:50:44.561465 3968 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:44.562870 3938 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:50:44.563073 3938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "bd079154587d4fe3be45bde9d465902c"
format_stamp: "Formatted at 2026-09-29 01:50:38 on dist-test-slave-vz23"
I20260929 01:50:44.563761 3938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:44.590950 3938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:44.591907 3938 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:44.592201 3938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:44.613233 3938 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:41041
I20260929 01:50:44.613260 4019 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:41041 every 8 connection(s)
I20260929 01:50:44.614851 3938 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:50:44.621559 4020 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:50:44.622525 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 3938
I20260929 01:50:44.622728 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:50:44.623937 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:39879
--webserver_interface=127.0.71.124
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:33125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879 with env {}
I20260929 01:50:44.626113 4020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c: Bootstrap starting.
I20260929 01:50:44.642876 4020 log.cc:826] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c: Log is configured to *not* fsync() on all Append() calls
I20260929 01:50:44.653540 4020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:50:44.654172 4020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c: Bootstrap complete.
I20260929 01:50:44.663380 4020 raft_consensus.cc:359] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } }
I20260929 01:50:44.664235 4020 raft_consensus.cc:740] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: bd079154587d4fe3be45bde9d465902c, State: Initialized, Role: FOLLOWER
I20260929 01:50:44.665074 4020 consensus_queue.cc:260] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } }
I20260929 01:50:44.665897 4024 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:44.666165 4024 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:44.667209 4020 sys_catalog.cc:565] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:50:44.677219 4035 catalog_manager.cc:1308] Loaded cluster ID: 59443e495098424cb36cef5e64d4d4bb
I20260929 01:50:44.677364 4035 catalog_manager.cc:1601] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c: loading cluster ID for follower catalog manager: success
I20260929 01:50:44.681671 4035 catalog_manager.cc:1623] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c: acquiring CA information for follower catalog manager: success
I20260929 01:50:44.684350 4035 catalog_manager.cc:1651] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:50:44.713289 3938 master_runner.cc:187] Error getting master registration for 127.0.71.124:39879: Network error: Client connection negotiation failed: client connection to 127.0.71.124:39879: connect: Connection refused (error 111),
I20260929 01:50:44.719172 3938 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:50:44.981273 4022 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:44.981662 4022 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:44.981794 4022 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:44.991089 4022 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:50:44.991268 4022 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:44.991364 4022 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:50:44.991460 4022 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:50:45.002257 4022 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:39879
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:45.004637 4022 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:45.007262 4022 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:45.017004 4043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:45.018958 4044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:45.021934 4046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:45.022555 4022 server_base.cc:1061] running on GCE node
I20260929 01:50:45.023355 4022 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:45.024942 4022 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:45.026156 4022 hybrid_clock.cc:648] HybridClock initialized: now 1790646645026117 us; error 40 us; skew 500 ppm
I20260929 01:50:45.026583 4022 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:45.028951 4022 webserver.cc:533] Webserver started at http://127.0.71.124:42663/ using document root <none> and password file <none>
I20260929 01:50:45.029718 4022 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:45.029850 4022 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:45.035631 4022 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 01:50:45.038880 4052 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:45.040500 4022 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 01:50:45.040679 4022 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "39a6fd98553d489881677bfc15bc95ff"
format_stamp: "Formatted at 2026-09-29 01:50:38 on dist-test-slave-vz23"
I20260929 01:50:45.041399 4022 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:45.074947 4022 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:45.075852 4022 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:45.076164 4022 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:45.097790 4022 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:39879
I20260929 01:50:45.097833 4103 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:39879 every 8 connection(s)
I20260929 01:50:45.099459 4022 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:50:45.103543 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 4022
I20260929 01:50:45.103722 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:50:45.107456 4104 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:50:45.112138 4104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff: Bootstrap starting.
I20260929 01:50:45.129895 4104 log.cc:826] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff: Log is configured to *not* fsync() on all Append() calls
I20260929 01:50:45.138173 4104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:50:45.138739 4104 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff: Bootstrap complete.
I20260929 01:50:45.145087 4104 raft_consensus.cc:359] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } }
I20260929 01:50:45.145751 4104 raft_consensus.cc:740] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 39a6fd98553d489881677bfc15bc95ff, State: Initialized, Role: FOLLOWER
I20260929 01:50:45.146395 4104 consensus_queue.cc:260] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } }
I20260929 01:50:45.147238 4109 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:45.147548 4109 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:45.148607 4104 sys_catalog.cc:565] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:50:45.159730 4120 catalog_manager.cc:1308] Loaded cluster ID: 59443e495098424cb36cef5e64d4d4bb
I20260929 01:50:45.159909 4120 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff: loading cluster ID for follower catalog manager: success
I20260929 01:50:45.163713 4120 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff: acquiring CA information for follower catalog manager: success
I20260929 01:50:45.166342 4120 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:50:45.187220 4022 master_runner.cc:264] No leader master found from master 39a6fd98553d489881677bfc15bc95ff
I20260929 01:50:45.187359 4022 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:50:45.448881 4123 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:50:45.449186 4123 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } }
I20260929 01:50:45.451226 4123 leader_election.cc:290] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers bd079154587d4fe3be45bde9d465902c (127.0.71.125:41041), 39a6fd98553d489881677bfc15bc95ff (127.0.71.124:39879)
I20260929 01:50:45.459929 3995 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7595c0707a67425096f7498511d0cbed" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bd079154587d4fe3be45bde9d465902c" is_pre_election: true
I20260929 01:50:45.460526 3995 raft_consensus.cc:2468] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7595c0707a67425096f7498511d0cbed in term 1.
I20260929 01:50:45.460601 4079 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7595c0707a67425096f7498511d0cbed" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39a6fd98553d489881677bfc15bc95ff" is_pre_election: true
I20260929 01:50:45.461171 4079 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7595c0707a67425096f7498511d0cbed in term 1.
I20260929 01:50:45.461153 3886 leader_election.cc:304] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7595c0707a67425096f7498511d0cbed, bd079154587d4fe3be45bde9d465902c; no voters:
I20260929 01:50:45.461632 4123 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 01:50:45.461807 4123 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:50:45.461916 4123 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:50:45.465480 4123 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } }
I20260929 01:50:45.466068 4123 leader_election.cc:290] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [CANDIDATE]: Term 2 election: Requested vote from peers bd079154587d4fe3be45bde9d465902c (127.0.71.125:41041), 39a6fd98553d489881677bfc15bc95ff (127.0.71.124:39879)
I20260929 01:50:45.466652 4079 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7595c0707a67425096f7498511d0cbed" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "39a6fd98553d489881677bfc15bc95ff"
I20260929 01:50:45.466637 3995 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7595c0707a67425096f7498511d0cbed" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "bd079154587d4fe3be45bde9d465902c"
I20260929 01:50:45.466835 3995 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:50:45.466876 4079 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:50:45.471318 4079 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7595c0707a67425096f7498511d0cbed in term 2.
I20260929 01:50:45.471817 3886 leader_election.cc:304] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 39a6fd98553d489881677bfc15bc95ff, 7595c0707a67425096f7498511d0cbed; no voters:
I20260929 01:50:45.471951 3995 raft_consensus.cc:2468] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7595c0707a67425096f7498511d0cbed in term 2.
I20260929 01:50:45.472158 4123 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:50:45.472698 4123 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [term 2 LEADER]: Becoming Leader. State: Replica: 7595c0707a67425096f7498511d0cbed, State: Running, Role: LEADER
I20260929 01:50:45.473263 4123 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } }
I20260929 01:50:45.476115 4126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7595c0707a67425096f7498511d0cbed. Latest consensus state: current_term: 2 leader_uuid: "7595c0707a67425096f7498511d0cbed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:45.476421 4126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:45.477190 4128 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:50:45.479146 4128 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:50:45.479925 4128 catalog_manager.cc:1308] Loaded cluster ID: 59443e495098424cb36cef5e64d4d4bb
I20260929 01:50:45.479992 4128 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:50:45.481143 4128 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:50:45.481988 4128 catalog_manager.cc:6230] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: Loaded TSK: 0
I20260929 01:50:45.483112 4128 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:50:45.512650 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--builtin_ntp_servers=127.0.71.84:33125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260929 01:50:45.851553 4129 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:45.851926 4129 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:45.852059 4129 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:45.860615 4129 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:45.860867 4129 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:50:45.871238 4129 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:45.872983 4129 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:45.874723 4129 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:45.884182 4135 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:45.884682 4137 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:45.884979 4134 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:45.885329 4079 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [term 2 FOLLOWER]: Refusing update from remote peer 7595c0707a67425096f7498511d0cbed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 01:50:45.885792 4129 server_base.cc:1061] running on GCE node
I20260929 01:50:45.886384 4129 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:45.886309 4126 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [LEADER]: Connected to new peer: Peer: permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:50:45.887465 4129 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:45.888671 4129 hybrid_clock.cc:648] HybridClock initialized: now 1790646645888613 us; error 47 us; skew 500 ppm
I20260929 01:50:45.889063 4129 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:45.890004 4140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7595c0707a67425096f7498511d0cbed. Latest consensus state: current_term: 2 leader_uuid: "7595c0707a67425096f7498511d0cbed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:45.890233 4140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:45.891649 4123 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "7595c0707a67425096f7498511d0cbed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:45.891929 4123 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:45.893183 4129 webserver.cc:533] Webserver started at http://127.0.71.65:34197/ using document root <none> and password file <none>
I20260929 01:50:45.893612 3995 raft_consensus.cc:1275] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [term 2 FOLLOWER]: Refusing update from remote peer 7595c0707a67425096f7498511d0cbed: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 01:50:45.894168 4129 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:45.894328 4129 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:45.894743 4140 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "7595c0707a67425096f7498511d0cbed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:45.894919 4140 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:45.895035 4146 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [LEADER]: Connected to new peer: Peer: permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.001s
I20260929 01:50:45.898921 4150 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "7595c0707a67425096f7498511d0cbed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:45.899255 4150 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:45.899727 4147 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7595c0707a67425096f7498511d0cbed. Latest consensus state: current_term: 2 leader_uuid: "7595c0707a67425096f7498511d0cbed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:45.900038 4147 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:45.901592 4129 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 01:50:45.904434 4152 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:45.905893 4129 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:50:45.906042 4129 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "4407206f6d574cfb95ac7d10f259df44"
format_stamp: "Formatted at 2026-09-29 01:50:39 on dist-test-slave-vz23"
I20260929 01:50:45.906690 4129 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:45.935503 4129 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:45.936362 4129 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:45.936704 4129 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:45.937636 4129 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:45.939280 4129 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:45.939357 4129 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:45.939441 4129 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:45.939484 4129 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:45.974560 4129 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:46053
I20260929 01:50:45.974651 4264 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:46053 every 8 connection(s)
I20260929 01:50:45.976111 4129 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:50:45.983428 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 4129
I20260929 01:50:45.983675 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
I20260929 01:50:45.989348 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.66:0
--local_ip_for_outbound_sockets=127.0.71.66
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--builtin_ntp_servers=127.0.71.84:33125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:50:46.019037 4266 heartbeater.cc:344] Connected to a master server at 127.0.71.126:40159
I20260929 01:50:46.019469 4266 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:46.020808 4266 heartbeater.cc:507] Master 127.0.71.126:40159 requested a full tablet report, sending...
I20260929 01:50:46.022765 4265 heartbeater.cc:344] Connected to a master server at 127.0.71.124:39879
I20260929 01:50:46.022903 4265 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:46.023305 4267 heartbeater.cc:344] Connected to a master server at 127.0.71.125:41041
I20260929 01:50:46.023458 4265 heartbeater.cc:507] Master 127.0.71.124:39879 requested a full tablet report, sending...
I20260929 01:50:46.023481 4267 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:46.024055 4267 heartbeater.cc:507] Master 127.0.71.125:41041 requested a full tablet report, sending...
I20260929 01:50:46.024192 3901 ts_manager.cc:194] Registered new tserver with Master: 4407206f6d574cfb95ac7d10f259df44 (127.0.71.65:46053)
I20260929 01:50:46.025192 4068 ts_manager.cc:194] Registered new tserver with Master: 4407206f6d574cfb95ac7d10f259df44 (127.0.71.65:46053)
I20260929 01:50:46.026594 3901 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:53823
I20260929 01:50:46.027024 3985 ts_manager.cc:194] Registered new tserver with Master: 4407206f6d574cfb95ac7d10f259df44 (127.0.71.65:46053)
W20260929 01:50:46.337244 4275 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:46.337543 4275 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:46.337613 4275 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260929 01:50:46.343667 4126 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "7595c0707a67425096f7498511d0cbed" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:46.343979 4126 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [sys.catalog]: This master's current role is: LEADER
W20260929 01:50:46.346377 4275 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:46.346547 4275 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.66
I20260929 01:50:46.357218 4275 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:46.358739 4275 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:46.360433 4275 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:46.370309 4280 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:46.370352 4281 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:46.371191 4283 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:46.372430 4275 server_base.cc:1061] running on GCE node
I20260929 01:50:46.373132 4275 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:46.374042 4275 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:46.375315 4275 hybrid_clock.cc:648] HybridClock initialized: now 1790646646375265 us; error 50 us; skew 500 ppm
I20260929 01:50:46.375643 4275 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:46.377875 4275 webserver.cc:533] Webserver started at http://127.0.71.66:39569/ using document root <none> and password file <none>
I20260929 01:50:46.378607 4275 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:46.378747 4275 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:46.384042 4275 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:50:46.386946 4289 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:46.388414 4275 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 01:50:46.388718 4275 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/wal
uuid: "0b15cef5b60b4e8db93620cd7d94d200"
format_stamp: "Formatted at 2026-09-29 01:50:40 on dist-test-slave-vz23"
I20260929 01:50:46.389473 4275 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:46.404382 4275 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:46.405404 4275 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:46.405725 4275 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:46.406721 4275 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:46.408703 4275 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:46.408818 4275 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:46.408969 4275 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:46.409046 4275 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:46.447310 4275 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.66:46623
I20260929 01:50:46.447358 4401 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.66:46623 every 8 connection(s)
I20260929 01:50:46.449097 4275 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
I20260929 01:50:46.456591 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 4275
I20260929 01:50:46.456773 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-1/wal/instance
I20260929 01:50:46.462146 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.67:0
--local_ip_for_outbound_sockets=127.0.71.67
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--builtin_ntp_servers=127.0.71.84:33125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:50:46.466748 4402 heartbeater.cc:344] Connected to a master server at 127.0.71.124:39879
I20260929 01:50:46.467170 4402 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:46.467509 4403 heartbeater.cc:344] Connected to a master server at 127.0.71.126:40159
I20260929 01:50:46.467655 4403 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:46.468477 4402 heartbeater.cc:507] Master 127.0.71.124:39879 requested a full tablet report, sending...
I20260929 01:50:46.468562 4403 heartbeater.cc:507] Master 127.0.71.126:40159 requested a full tablet report, sending...
I20260929 01:50:46.469480 4404 heartbeater.cc:344] Connected to a master server at 127.0.71.125:41041
I20260929 01:50:46.469640 4404 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:46.470088 4404 heartbeater.cc:507] Master 127.0.71.125:41041 requested a full tablet report, sending...
I20260929 01:50:46.470314 3901 ts_manager.cc:194] Registered new tserver with Master: 0b15cef5b60b4e8db93620cd7d94d200 (127.0.71.66:46623)
I20260929 01:50:46.471045 4068 ts_manager.cc:194] Registered new tserver with Master: 0b15cef5b60b4e8db93620cd7d94d200 (127.0.71.66:46623)
I20260929 01:50:46.471341 3985 ts_manager.cc:194] Registered new tserver with Master: 0b15cef5b60b4e8db93620cd7d94d200 (127.0.71.66:46623)
I20260929 01:50:46.471386 3901 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.66:37113
W20260929 01:50:46.794664 4409 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:46.794994 4409 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:46.795110 4409 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:46.803279 4409 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:46.803483 4409 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.67
I20260929 01:50:46.813745 4409 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:46.815264 4409 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:46.816970 4409 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:46.826695 4415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:46.826717 4414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:46.826925 4417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:46.828240 4409 server_base.cc:1061] running on GCE node
I20260929 01:50:46.828910 4409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:46.829914 4409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:46.831148 4409 hybrid_clock.cc:648] HybridClock initialized: now 1790646646831100 us; error 42 us; skew 500 ppm
I20260929 01:50:46.831521 4409 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:46.833763 4409 webserver.cc:533] Webserver started at http://127.0.71.67:41943/ using document root <none> and password file <none>
I20260929 01:50:46.834491 4409 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:46.834591 4409 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:46.839875 4409 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:50:46.842720 4423 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:46.844553 4409 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:50:46.844712 4409 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/wal
uuid: "e5477c1a84e244019033fc3ea62d98b6"
format_stamp: "Formatted at 2026-09-29 01:50:40 on dist-test-slave-vz23"
I20260929 01:50:46.845409 4409 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:46.862429 4409 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:46.863260 4409 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:46.863511 4409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:46.864544 4409 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:46.866132 4409 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:46.866204 4409 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:46.866310 4409 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:46.866359 4409 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:46.902925 4409 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.67:35623
I20260929 01:50:46.903064 4535 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.67:35623 every 8 connection(s)
I20260929 01:50:46.904536 4409 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
I20260929 01:50:46.905357 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 4409
I20260929 01:50:46.905566 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-2/wal/instance
I20260929 01:50:46.935293 4536 heartbeater.cc:344] Connected to a master server at 127.0.71.124:39879
I20260929 01:50:46.935719 4536 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:46.935998 4537 heartbeater.cc:344] Connected to a master server at 127.0.71.126:40159
I20260929 01:50:46.936146 4537 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:46.936569 4538 heartbeater.cc:344] Connected to a master server at 127.0.71.125:41041
I20260929 01:50:46.936733 4538 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:46.937026 4536 heartbeater.cc:507] Master 127.0.71.124:39879 requested a full tablet report, sending...
I20260929 01:50:46.937088 4537 heartbeater.cc:507] Master 127.0.71.126:40159 requested a full tablet report, sending...
I20260929 01:50:46.937233 4538 heartbeater.cc:507] Master 127.0.71.125:41041 requested a full tablet report, sending...
I20260929 01:50:46.938902 3900 ts_manager.cc:194] Registered new tserver with Master: e5477c1a84e244019033fc3ea62d98b6 (127.0.71.67:35623)
I20260929 01:50:46.938958 4068 ts_manager.cc:194] Registered new tserver with Master: e5477c1a84e244019033fc3ea62d98b6 (127.0.71.67:35623)
I20260929 01:50:46.939610 3984 ts_manager.cc:194] Registered new tserver with Master: e5477c1a84e244019033fc3ea62d98b6 (127.0.71.67:35623)
I20260929 01:50:46.940644 3900 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.67:58257
I20260929 01:50:46.953562 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:50:46.961735 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:50:46.961920 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 3870
I20260929 01:50:46.962882 3931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:47.029201 4266 heartbeater.cc:499] Master 127.0.71.126:40159 was elected leader, sending a full tablet report...
I20260929 01:50:47.085021 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 3870
I20260929 01:50:47.107394 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:40159
--webserver_interface=127.0.71.126
--webserver_port=44505
--builtin_ntp_servers=127.0.71.84:33125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879 with env {}
W20260929 01:50:47.467497 4544 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:47.467820 4544 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:47.467890 4544 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260929 01:50:47.473479 4403 heartbeater.cc:499] Master 127.0.71.126:40159 was elected leader, sending a full tablet report...
W20260929 01:50:47.475024 4403 heartbeater.cc:646] Failed to heartbeat to 127.0.71.126:40159 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.71.126:40159: connect: Connection refused (error 111)
W20260929 01:50:47.477481 4544 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:50:47.477567 4544 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:47.477625 4544 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:50:47.477665 4544 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:50:47.487918 4544 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:40159
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=44505
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:47.489575 4544 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:47.491254 4544 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:47.499477 4551 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:47.499439 4550 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:47.500267 4553 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:47.500479 4544 server_base.cc:1061] running on GCE node
I20260929 01:50:47.501374 4544 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:47.503501 4544 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:47.504710 4544 hybrid_clock.cc:648] HybridClock initialized: now 1790646647504639 us; error 64 us; skew 500 ppm
I20260929 01:50:47.505159 4544 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:47.507546 4544 webserver.cc:533] Webserver started at http://127.0.71.126:44505/ using document root <none> and password file <none>
I20260929 01:50:47.508275 4544 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:47.508392 4544 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:47.514040 4544 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:50:47.517009 4559 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:47.518596 4544 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 01:50:47.518826 4544 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "7595c0707a67425096f7498511d0cbed"
format_stamp: "Formatted at 2026-09-29 01:50:37 on dist-test-slave-vz23"
I20260929 01:50:47.519632 4544 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:47.556556 4544 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:47.557533 4544 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:47.557881 4544 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:47.581112 4544 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:40159
I20260929 01:50:47.581136 4610 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:40159 every 8 connection(s)
I20260929 01:50:47.582895 4544 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:50:47.587900 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 4544
I20260929 01:50:47.588196 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 4129
I20260929 01:50:47.591176 4611 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:50:47.595628 4611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: Bootstrap starting.
I20260929 01:50:47.599108 4260 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:47.621306 4611 log.cc:826] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: Log is configured to *not* fsync() on all Append() calls
I20260929 01:50:47.633133 4611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:50:47.633786 4611 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: Bootstrap complete.
I20260929 01:50:47.644842 4611 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } }
I20260929 01:50:47.645870 4611 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7595c0707a67425096f7498511d0cbed, State: Initialized, Role: FOLLOWER
I20260929 01:50:47.646790 4611 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } }
I20260929 01:50:47.647716 4617 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:47.648033 4617 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:47.649112 4611 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:50:47.659881 4628 catalog_manager.cc:1308] Loaded cluster ID: 59443e495098424cb36cef5e64d4d4bb
I20260929 01:50:47.660082 4628 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: loading cluster ID for follower catalog manager: success
I20260929 01:50:47.664258 4628 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: acquiring CA information for follower catalog manager: success
I20260929 01:50:47.666859 4628 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:50:47.707508 4544 master_runner.cc:264] No leader master found from master 7595c0707a67425096f7498511d0cbed
I20260929 01:50:47.707707 4544 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:50:47.731467 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 4129
I20260929 01:50:47.748803 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:46053
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--webserver_port=34197
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:33125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:50:47.942739 4537 heartbeater.cc:499] Master 127.0.71.126:40159 was elected leader, sending a full tablet report...
I20260929 01:50:47.954710 4576 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "e5477c1a84e244019033fc3ea62d98b6" instance_seqno: 1790646646894361) as {username='slave'} at 127.0.71.67:39453; Asking this server to re-register.
I20260929 01:50:47.955714 4537 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:47.957616 4576 ts_manager.cc:194] Registered new tserver with Master: e5477c1a84e244019033fc3ea62d98b6 (127.0.71.67:35623)
W20260929 01:50:48.092475 4634 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:48.092796 4634 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:48.092880 4634 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:48.102007 4634 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:48.102228 4634 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:50:48.112393 4634 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:46053
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=34197
--tserver_master_addrs=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:48.114094 4634 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:48.115901 4634 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:48.126603 4640 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:48.126564 4641 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:48.126638 4643 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:48.127594 4634 server_base.cc:1061] running on GCE node
I20260929 01:50:48.128296 4634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:48.129343 4634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:48.130558 4634 hybrid_clock.cc:648] HybridClock initialized: now 1790646648130518 us; error 26 us; skew 500 ppm
I20260929 01:50:48.130951 4634 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:48.133242 4634 webserver.cc:533] Webserver started at http://127.0.71.65:34197/ using document root <none> and password file <none>
I20260929 01:50:48.133986 4634 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:48.134079 4634 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:48.139667 4634 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:50:48.142761 4649 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:48.144299 4634 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:50:48.144544 4634 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "4407206f6d574cfb95ac7d10f259df44"
format_stamp: "Formatted at 2026-09-29 01:50:39 on dist-test-slave-vz23"
I20260929 01:50:48.145366 4634 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:48.164489 4634 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:48.165397 4634 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:48.165693 4634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:48.166630 4634 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:48.168416 4634 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:48.168558 4634 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 01:50:48.168730 4634 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:48.168808 4634 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:48.203531 4634 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:46053
I20260929 01:50:48.203550 4761 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:46053 every 8 connection(s)
I20260929 01:50:48.205281 4634 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:50:48.208109 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 4634
I20260929 01:50:48.208396 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 4634
I20260929 01:50:48.223359 4763 heartbeater.cc:344] Connected to a master server at 127.0.71.126:40159
I20260929 01:50:48.223809 4763 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:48.224933 4763 heartbeater.cc:507] Master 127.0.71.126:40159 requested a full tablet report, sending...
I20260929 01:50:48.226505 4757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:48.227418 4576 ts_manager.cc:194] Registered new tserver with Master: 4407206f6d574cfb95ac7d10f259df44 (127.0.71.65:46053)
I20260929 01:50:48.277621 4774 raft_consensus.cc:493] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7595c0707a67425096f7498511d0cbed)
I20260929 01:50:48.277837 4774 raft_consensus.cc:515] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } }
I20260929 01:50:48.279875 4774 leader_election.cc:290] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7595c0707a67425096f7498511d0cbed (127.0.71.126:40159), 39a6fd98553d489881677bfc15bc95ff (127.0.71.124:39879)
I20260929 01:50:48.287324 4079 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd079154587d4fe3be45bde9d465902c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39a6fd98553d489881677bfc15bc95ff" is_pre_election: true
I20260929 01:50:48.288136 4586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd079154587d4fe3be45bde9d465902c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7595c0707a67425096f7498511d0cbed" is_pre_election: true
I20260929 01:50:48.288642 4586 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate bd079154587d4fe3be45bde9d465902c in term 2.
I20260929 01:50:48.289171 3971 leader_election.cc:304] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7595c0707a67425096f7498511d0cbed, bd079154587d4fe3be45bde9d465902c; no voters: 39a6fd98553d489881677bfc15bc95ff
I20260929 01:50:48.289649 4774 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 01:50:48.289813 4774 raft_consensus.cc:493] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7595c0707a67425096f7498511d0cbed)
I20260929 01:50:48.290023 4774 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:50:48.292359 4774 raft_consensus.cc:515] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } }
I20260929 01:50:48.293108 4774 leader_election.cc:290] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [CANDIDATE]: Term 3 election: Requested vote from peers 7595c0707a67425096f7498511d0cbed (127.0.71.126:40159), 39a6fd98553d489881677bfc15bc95ff (127.0.71.124:39879)
I20260929 01:50:48.293602 4586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd079154587d4fe3be45bde9d465902c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7595c0707a67425096f7498511d0cbed"
I20260929 01:50:48.293715 4079 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "bd079154587d4fe3be45bde9d465902c" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "39a6fd98553d489881677bfc15bc95ff"
I20260929 01:50:48.293841 4586 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:50:48.298215 4586 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate bd079154587d4fe3be45bde9d465902c in term 3.
I20260929 01:50:48.298707 3971 leader_election.cc:304] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7595c0707a67425096f7498511d0cbed, bd079154587d4fe3be45bde9d465902c; no voters: 39a6fd98553d489881677bfc15bc95ff
I20260929 01:50:48.299067 4774 raft_consensus.cc:2804] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [term 3 FOLLOWER]: Leader election won for term 3
I20260929 01:50:48.299521 4774 raft_consensus.cc:697] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [term 3 LEADER]: Becoming Leader. State: Replica: bd079154587d4fe3be45bde9d465902c, State: Running, Role: LEADER
I20260929 01:50:48.299993 4774 consensus_queue.cc:237] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } }
I20260929 01:50:48.301939 4777 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd079154587d4fe3be45bde9d465902c. Latest consensus state: current_term: 3 leader_uuid: "bd079154587d4fe3be45bde9d465902c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:48.302234 4777 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:48.306841 4779 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:50:48.308817 4779 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:50:48.309579 4779 catalog_manager.cc:1308] Loaded cluster ID: 59443e495098424cb36cef5e64d4d4bb
I20260929 01:50:48.309644 4779 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:50:48.310904 4779 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:50:48.311774 4779 catalog_manager.cc:6230] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c: Loaded TSK: 0
I20260929 01:50:48.312754 4779 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:50:48.322971 4762 heartbeater.cc:344] Connected to a master server at 127.0.71.124:39879
I20260929 01:50:48.323576 4762 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:48.324208 4762 heartbeater.cc:507] Master 127.0.71.124:39879 requested a full tablet report, sending...
I20260929 01:50:48.324759 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 4634
I20260929 01:50:48.326275 4068 ts_manager.cc:194] Re-registered known tserver with Master: 4407206f6d574cfb95ac7d10f259df44 (127.0.71.65:46053)
W20260929 01:50:48.341661 4053 connection.cc:573] server connection from 127.0.71.65:46519 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:50:48.342628 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:46053
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--webserver_port=34197
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:33125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {}
W20260929 01:50:48.342938 4771 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:50:48.211213 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:47279 (local address 127.0.71.125:41041)
0929 01:50:48.215627 (+ 4414us) server_negotiation.cc:207] Beginning negotiation
0929 01:50:48.215635 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0929 01:50:48.215658 (+ 23us) server_negotiation.cc:408] Connection header received
0929 01:50:48.215724 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:50:48.215730 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:50:48.215808 (+ 78us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:50:48.215930 (+ 122us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:50:48.220332 (+ 4402us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:50:48.221762 (+ 1430us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:50:48.226849 (+ 5087us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:50:48.227270 (+ 421us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0929 01:50:48.324880 (+ 97610us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0929 01:50:48.324956 (+ 76us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0929 01:50:48.324965 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: PLAIN
0929 01:50:48.324989 (+ 24us) server_negotiation.cc:923] calling sasl_server_start()
0929 01:50:48.325040 (+ 51us) server_negotiation.cc:1092] Received PLAIN auth, user=slave
0929 01:50:48.325060 (+ 20us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0929 01:50:48.325152 (+ 92us) server_negotiation.cc:1036] Waiting for connection context
0929 01:50:48.341685 (+ 16533us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:47279: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":4297,"thread_start_us":4203,"threads_started":1}
I20260929 01:50:48.369998 4781 raft_consensus.cc:493] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7595c0707a67425096f7498511d0cbed)
I20260929 01:50:48.370234 4781 raft_consensus.cc:515] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } }
I20260929 01:50:48.372159 4781 leader_election.cc:290] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7595c0707a67425096f7498511d0cbed (127.0.71.126:40159), bd079154587d4fe3be45bde9d465902c (127.0.71.125:41041)
I20260929 01:50:48.372861 3995 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "39a6fd98553d489881677bfc15bc95ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "bd079154587d4fe3be45bde9d465902c" is_pre_election: true
I20260929 01:50:48.380123 4586 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "39a6fd98553d489881677bfc15bc95ff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7595c0707a67425096f7498511d0cbed" is_pre_election: true
I20260929 01:50:48.380555 4586 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 39a6fd98553d489881677bfc15bc95ff in current term 3: Already voted for candidate bd079154587d4fe3be45bde9d465902c in this term.
I20260929 01:50:48.381457 4055 leader_election.cc:304] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 39a6fd98553d489881677bfc15bc95ff; no voters: 7595c0707a67425096f7498511d0cbed, bd079154587d4fe3be45bde9d465902c
I20260929 01:50:48.381896 4781 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:50:48.384089 4781 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 01:50:48.482072 4403 heartbeater.cc:344] Connected to a master server at 127.0.71.126:40159
I20260929 01:50:48.482363 4403 heartbeater.cc:499] Master 127.0.71.126:40159 was elected leader, sending a full tablet report...
I20260929 01:50:48.483304 4576 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "0b15cef5b60b4e8db93620cd7d94d200" instance_seqno: 1790646646438752) as {username='slave'} at 127.0.71.66:35237; Asking this server to re-register.
I20260929 01:50:48.483985 4403 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:48.485083 4576 ts_manager.cc:194] Registered new tserver with Master: 0b15cef5b60b4e8db93620cd7d94d200 (127.0.71.66:46623)
W20260929 01:50:48.684154 4780 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:48.684566 4780 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:48.684685 4780 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:50:48.684803 4780 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:48.693194 4780 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:48.693440 4780 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:50:48.703711 4780 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:46053
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=34197
--tserver_master_addrs=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:48.705410 4780 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:48.707181 4780 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260929 01:50:48.712811 4079 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [term 3 FOLLOWER]: Refusing update from remote peer bd079154587d4fe3be45bde9d465902c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 01:50:48.713562 4777 consensus_queue.cc:1048] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [LEADER]: Connected to new peer: Peer: permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:50:48.716506 4586 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [term 3 FOLLOWER]: Refusing update from remote peer bd079154587d4fe3be45bde9d465902c: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 01:50:48.717480 4777 consensus_queue.cc:1048] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [LEADER]: Connected to new peer: Peer: permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260929 01:50:48.720407 4789 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:48.729969 4791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd079154587d4fe3be45bde9d465902c. Latest consensus state: current_term: 3 leader_uuid: "bd079154587d4fe3be45bde9d465902c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:48.730400 4791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:48.730471 4781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [sys.catalog]: SysCatalogTable state changed. Reason: New leader bd079154587d4fe3be45bde9d465902c. Latest consensus state: current_term: 3 leader_uuid: "bd079154587d4fe3be45bde9d465902c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:48.730830 4781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:48.732488 4777 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "bd079154587d4fe3be45bde9d465902c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:48.732720 4777 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:48.733338 4777 sys_catalog.cc:455] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "bd079154587d4fe3be45bde9d465902c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:48.733333 4781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "bd079154587d4fe3be45bde9d465902c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:48.733500 4777 sys_catalog.cc:458] T 00000000000000000000000000000000 P bd079154587d4fe3be45bde9d465902c [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:48.733522 4781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 39a6fd98553d489881677bfc15bc95ff [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:50:48.735054 4790 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:48.736744 4791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "bd079154587d4fe3be45bde9d465902c" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7595c0707a67425096f7498511d0cbed" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40159 } } peers { permanent_uuid: "bd079154587d4fe3be45bde9d465902c" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 41041 } } peers { permanent_uuid: "39a6fd98553d489881677bfc15bc95ff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39879 } } }
I20260929 01:50:48.736945 4791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7595c0707a67425096f7498511d0cbed [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:50:48.737418 4796 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:48.737645 4780 server_base.cc:1061] running on GCE node
I20260929 01:50:48.738307 4780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:48.739359 4780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:48.740626 4780 hybrid_clock.cc:648] HybridClock initialized: now 1790646648740546 us; error 49 us; skew 500 ppm
I20260929 01:50:48.741057 4780 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:48.743637 4780 webserver.cc:533] Webserver started at http://127.0.71.65:34197/ using document root <none> and password file <none>
I20260929 01:50:48.744642 4780 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:48.744817 4780 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:48.750828 4780 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:50:48.753931 4803 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:48.755573 4780 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:50:48.755836 4780 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "4407206f6d574cfb95ac7d10f259df44"
format_stamp: "Formatted at 2026-09-29 01:50:39 on dist-test-slave-vz23"
I20260929 01:50:48.756708 4780 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:48.781713 4780 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:48.782804 4780 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:48.783113 4780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:48.784176 4780 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:48.786074 4780 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:48.786213 4780 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:48.786368 4780 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:48.786468 4780 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:50:48.786970 4780 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:50:48.787088 4780 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:50:48.787166 4780 fault_injection.cc:59] injecting fault for kudu (pid 4780): FLAGS_fault_before_start (process will exit)
I20260929 01:50:48.803530 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:46053
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--webserver_port=34197
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:33125
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260929 01:50:49.146807 4811 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:49.147222 4811 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:49.147337 4811 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:49.156149 4811 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:49.156319 4811 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:50:49.166525 4811 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33125
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:46053
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=34197
--tserver_master_addrs=127.0.71.126:40159,127.0.71.125:41041,127.0.71.124:39879
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:49.168116 4811 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:49.170126 4811 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:49.180431 4817 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:49.180666 4819 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:49.180339 4816 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:49.181679 4811 server_base.cc:1061] running on GCE node
I20260929 01:50:49.182235 4811 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:49.183111 4811 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:49.184332 4811 hybrid_clock.cc:648] HybridClock initialized: now 1790646649184249 us; error 61 us; skew 500 ppm
I20260929 01:50:49.184674 4811 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:49.186757 4811 webserver.cc:533] Webserver started at http://127.0.71.65:34197/ using document root <none> and password file <none>
I20260929 01:50:49.187496 4811 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:49.187610 4811 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:49.193068 4811 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:50:49.195978 4825 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:49.197723 4811 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:50:49.197906 4811 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "4407206f6d574cfb95ac7d10f259df44"
format_stamp: "Formatted at 2026-09-29 01:50:39 on dist-test-slave-vz23"
I20260929 01:50:49.198608 4811 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:49.210675 4811 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:49.211561 4811 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:49.211853 4811 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:49.212819 4811 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:49.214568 4811 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:49.214638 4811 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:49.214787 4811 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:49.214838 4811 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:49.250783 4811 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:46053
I20260929 01:50:49.250818 4937 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:46053 every 8 connection(s)
I20260929 01:50:49.252835 4811 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:50:49.262338 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 4811
I20260929 01:50:49.262645 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 4811
I20260929 01:50:49.271416 4939 heartbeater.cc:344] Connected to a master server at 127.0.71.124:39879
I20260929 01:50:49.271884 4939 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:49.272332 4941 heartbeater.cc:344] Connected to a master server at 127.0.71.125:41041
I20260929 01:50:49.272553 4941 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:49.272837 4940 heartbeater.cc:344] Connected to a master server at 127.0.71.126:40159
I20260929 01:50:49.273402 4940 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:49.273898 4939 heartbeater.cc:507] Master 127.0.71.124:39879 requested a full tablet report, sending...
I20260929 01:50:49.274739 4941 heartbeater.cc:507] Master 127.0.71.125:41041 requested a full tablet report, sending...
I20260929 01:50:49.274791 4940 heartbeater.cc:507] Master 127.0.71.126:40159 requested a full tablet report, sending...
I20260929 01:50:49.275882 4068 ts_manager.cc:194] Re-registered known tserver with Master: 4407206f6d574cfb95ac7d10f259df44 (127.0.71.65:46053)
I20260929 01:50:49.276645 3984 ts_manager.cc:194] Re-registered known tserver with Master: 4407206f6d574cfb95ac7d10f259df44 (127.0.71.65:46053)
I20260929 01:50:49.277714 4576 ts_manager.cc:194] Re-registered known tserver with Master: 4407206f6d574cfb95ac7d10f259df44 (127.0.71.65:46053)
I20260929 01:50:49.278010 3984 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:43091
I20260929 01:50:49.288967 4933 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:49.380645 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 4811
I20260929 01:50:49.397185 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 4275
I20260929 01:50:49.405778 4397 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:49.492302 4404 heartbeater.cc:499] Master 127.0.71.125:41041 was elected leader, sending a full tablet report...
I20260929 01:50:49.494609 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 4275
I20260929 01:50:49.510722 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 4409
I20260929 01:50:49.518795 4531 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:49.612138 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 4409
I20260929 01:50:49.629158 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 4544
I20260929 01:50:49.635812 4606 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:49.741592 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 4544
W20260929 01:50:49.762483 3971 connection.cc:573] client connection to 127.0.71.126:40159 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:50:49.762554 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 3938
I20260929 01:50:49.762719 3938 master_runner.cc:217] Error getting master consensus for 127.0.71.126:40159: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:50:49.762946 3938 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:50:49.763746 4015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:49.871713 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 3938
I20260929 01:50:49.891830 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 4022
I20260929 01:50:49.893208 4099 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:49.999734 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 4022
2026-09-29T01:50:50Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestBasicOperation/1 (12764 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/2
2026-09-29T01:50:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:50:50Z Disabled control of system clock
2026-09-29T01:50:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:50:50Z Disabled control of system clock
2026-09-29T01:50:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:50:50Z Disabled control of system clock
2026-09-29T01:50:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:50:50Z Disabled control of system clock
2026-09-29T01:50:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:50:50Z Disabled control of system clock
I20260929 01:50:50.146473 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:43209
--webserver_interface=127.0.71.126
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:38621,127.0.71.85:33511,127.0.71.86:38913,127.0.71.87:45783,127.0.71.88:43239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453 with env {}
W20260929 01:50:50.478804 4971 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:50.479187 4971 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:50.479318 4971 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:50.487665 4971 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:50:50.487794 4971 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:50.487916 4971 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:50:50.488006 4971 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:50:50.498212 4971 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38621,127.0.71.85:33511,127.0.71.86:38913,127.0.71.87:45783,127.0.71.88:43239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:43209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:50.499897 4971 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:50.501750 4971 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:50.510156 4976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:50.510188 4977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:50.510614 4979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:50.511184 4971 server_base.cc:1061] running on GCE node
I20260929 01:50:50.512017 4971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:50.513732 4971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:50.515084 4971 hybrid_clock.cc:648] HybridClock initialized: now 1790646650514962 us; error 86 us; skew 500 ppm
I20260929 01:50:50.515489 4971 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:50.517884 4971 webserver.cc:533] Webserver started at http://127.0.71.126:35505/ using document root <none> and password file <none>
I20260929 01:50:50.518612 4971 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:50.518746 4971 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:50.519098 4971 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:50:50.521507 4971 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "4bea367d8bb04af49f22eaafc8bd4d03"
format_stamp: "Formatted at 2026-09-29 01:50:50 on dist-test-slave-vz23"
I20260929 01:50:50.522270 4971 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "4bea367d8bb04af49f22eaafc8bd4d03"
format_stamp: "Formatted at 2026-09-29 01:50:50 on dist-test-slave-vz23"
I20260929 01:50:50.527518 4971 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:50:50.531131 4985 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:50.532675 4971 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:50:50.532859 4971 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "4bea367d8bb04af49f22eaafc8bd4d03"
format_stamp: "Formatted at 2026-09-29 01:50:50 on dist-test-slave-vz23"
I20260929 01:50:50.533079 4971 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:50.555045 4971 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:50.555918 4971 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:50.556202 4971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:50.577418 4971 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:43209
I20260929 01:50:50.577450 5036 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:43209 every 8 connection(s)
I20260929 01:50:50.578965 4971 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:50:50.583156 5037 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:50:50.583824 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 4971
I20260929 01:50:50.584024 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:50:50.585402 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:42769
--webserver_interface=127.0.71.125
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:38621,127.0.71.85:33511,127.0.71.86:38913,127.0.71.87:45783,127.0.71.88:43239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453 with env {}
I20260929 01:50:50.591480 5037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:50:50.614804 5037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:50:50.616218 4987 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.125:42769: connect: Connection refused (error 111)
W20260929 01:50:50.617961 5037 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:42769: Network error: Client connection negotiation failed: client connection to 127.0.71.125:42769: connect: Connection refused (error 111)
I20260929 01:50:50.667340 5037 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } attempt: 1
W20260929 01:50:50.669677 5037 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:42769: Network error: Client connection negotiation failed: client connection to 127.0.71.125:42769: connect: Connection refused (error 111)
I20260929 01:50:50.737913 5037 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } attempt: 2
W20260929 01:50:50.740226 5037 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:42769: Network error: Client connection negotiation failed: client connection to 127.0.71.125:42769: connect: Connection refused (error 111)
I20260929 01:50:50.831494 5037 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } attempt: 3
W20260929 01:50:50.833796 5037 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:42769: Network error: Client connection negotiation failed: client connection to 127.0.71.125:42769: connect: Connection refused (error 111)
W20260929 01:50:50.926174 5039 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:50.926541 5039 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:50.926677 5039 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:50.935688 5039 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:50:50.935794 5039 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:50.935896 5039 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:50:50.935971 5039 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:50:50.946451 5039 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38621,127.0.71.85:33511,127.0.71.86:38913,127.0.71.87:45783,127.0.71.88:43239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:42769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:50.948180 5039 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:50.950067 5039 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:50.958576 5047 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:50.958611 5048 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:50.959159 5050 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:50.960059 5039 server_base.cc:1061] running on GCE node
I20260929 01:50:50.960875 5039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:50.962438 5039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:50.963644 5039 hybrid_clock.cc:648] HybridClock initialized: now 1790646650963481 us; error 134 us; skew 500 ppm
I20260929 01:50:50.964022 5039 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:50.966184 5039 webserver.cc:533] Webserver started at http://127.0.71.125:42949/ using document root <none> and password file <none>
I20260929 01:50:50.966920 5039 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:50.967013 5039 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:50.967376 5039 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:50:50.969794 5039 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/data/instance:
uuid: "32119a63667f414c9e4851a7a4a5db64"
format_stamp: "Formatted at 2026-09-29 01:50:50 on dist-test-slave-vz23"
I20260929 01:50:50.970562 5039 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/wal/instance:
uuid: "32119a63667f414c9e4851a7a4a5db64"
format_stamp: "Formatted at 2026-09-29 01:50:50 on dist-test-slave-vz23"
I20260929 01:50:50.975945 5039 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:50:50.977080 5037 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } attempt: 4
W20260929 01:50:50.979104 5037 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:42769: Network error: Client connection negotiation failed: client connection to 127.0.71.125:42769: connect: Connection refused (error 111)
I20260929 01:50:50.979933 5056 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:50.981359 5039 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:50:50.981564 5039 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "32119a63667f414c9e4851a7a4a5db64"
format_stamp: "Formatted at 2026-09-29 01:50:50 on dist-test-slave-vz23"
I20260929 01:50:50.981774 5039 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:51.010300 5039 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:51.011235 5039 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:51.011525 5039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:51.031517 5039 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:42769
I20260929 01:50:51.031632 5107 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:42769 every 8 connection(s)
I20260929 01:50:51.033370 5039 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:50:51.035458 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 5039
I20260929 01:50:51.035632 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:50:51.036796 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:38453
--webserver_interface=127.0.71.124
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:38621,127.0.71.85:33511,127.0.71.86:38913,127.0.71.87:45783,127.0.71.88:43239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453 with env {}
I20260929 01:50:51.037923 5108 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:50:51.044324 5108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:50:51.058892 5108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:50:51.069082 5108 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:50:51.070257 5059 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.124:38453: connect: Connection refused (error 111)
W20260929 01:50:51.071861 5108 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:38453: Network error: Client connection negotiation failed: client connection to 127.0.71.124:38453: connect: Connection refused (error 111)
I20260929 01:50:51.121172 5108 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } attempt: 1
W20260929 01:50:51.123234 5108 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:38453: Network error: Client connection negotiation failed: client connection to 127.0.71.124:38453: connect: Connection refused (error 111)
I20260929 01:50:51.191516 5108 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } attempt: 2
W20260929 01:50:51.193531 5108 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:38453: Network error: Client connection negotiation failed: client connection to 127.0.71.124:38453: connect: Connection refused (error 111)
I20260929 01:50:51.278366 5037 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } attempt: 5
I20260929 01:50:51.284765 5108 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } attempt: 3
I20260929 01:50:51.284919 5037 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:50:51.286701 5108 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:38453: Network error: Client connection negotiation failed: client connection to 127.0.71.124:38453: connect: Connection refused (error 111)
W20260929 01:50:51.287107 5037 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:38453: Network error: Client connection negotiation failed: client connection to 127.0.71.124:38453: connect: Connection refused (error 111)
I20260929 01:50:51.338362 5037 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } attempt: 1
W20260929 01:50:51.340341 5037 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:38453: Network error: Client connection negotiation failed: client connection to 127.0.71.124:38453: connect: Connection refused (error 111)
W20260929 01:50:51.374486 5110 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:51.374809 5110 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:51.374883 5110 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:51.383396 5110 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:50:51.383498 5110 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:51.383591 5110 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:50:51.383670 5110 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:50:51.394248 5110 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38621,127.0.71.85:33511,127.0.71.86:38913,127.0.71.87:45783,127.0.71.88:43239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:38453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:51.395922 5110 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:51.397884 5110 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:51.406512 5119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:51.406533 5118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:51.407793 5121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:51.407991 5110 server_base.cc:1061] running on GCE node
I20260929 01:50:51.408658 5037 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } attempt: 2
I20260929 01:50:51.408921 5110 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20260929 01:50:51.410521 5037 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:38453: Network error: Client connection negotiation failed: client connection to 127.0.71.124:38453: connect: Connection refused (error 111)
I20260929 01:50:51.410737 5110 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:51.411945 5110 hybrid_clock.cc:648] HybridClock initialized: now 1790646651411832 us; error 81 us; skew 500 ppm
I20260929 01:50:51.412322 5110 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:51.414642 5110 webserver.cc:533] Webserver started at http://127.0.71.124:36659/ using document root <none> and password file <none>
I20260929 01:50:51.415354 5110 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:51.415445 5110 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:51.415750 5110 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:50:51.418205 5110 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/data/instance:
uuid: "6c786aa789c24cdf89db361dfe2a28e1"
format_stamp: "Formatted at 2026-09-29 01:50:51 on dist-test-slave-vz23"
I20260929 01:50:51.418998 5110 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/wal/instance:
uuid: "6c786aa789c24cdf89db361dfe2a28e1"
format_stamp: "Formatted at 2026-09-29 01:50:51 on dist-test-slave-vz23"
I20260929 01:50:51.425551 5110 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 01:50:51.429055 5127 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:51.429929 5108 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } attempt: 4
I20260929 01:50:51.430621 5110 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 01:50:51.430804 5110 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "6c786aa789c24cdf89db361dfe2a28e1"
format_stamp: "Formatted at 2026-09-29 01:50:51 on dist-test-slave-vz23"
I20260929 01:50:51.431012 5110 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260929 01:50:51.431802 5108 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:38453: Network error: Client connection negotiation failed: client connection to 127.0.71.124:38453: connect: Connection refused (error 111)
I20260929 01:50:51.475382 5110 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:51.476302 5110 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:51.476625 5110 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:51.498272 5110 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:38453
I20260929 01:50:51.498286 5178 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:38453 every 8 connection(s)
I20260929 01:50:51.499817 5110 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:50:51.504343 5179 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:50:51.505973 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 5110
I20260929 01:50:51.506173 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:50:51.512825 5179 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:50:51.516760 5037 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } attempt: 3
I20260929 01:50:51.527364 5179 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:50:51.533250 5037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: Bootstrap starting.
I20260929 01:50:51.536821 5179 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:50:51.537354 5037 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: Neither blocks nor log segments found. Creating new log.
I20260929 01:50:51.538612 5037 log.cc:826] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: Log is configured to *not* fsync() on all Append() calls
I20260929 01:50:51.541256 5037 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: No bootstrap required, opened a new log
I20260929 01:50:51.547261 5037 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:50:51.547541 5037 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:50:51.547680 5037 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea367d8bb04af49f22eaafc8bd4d03, State: Initialized, Role: FOLLOWER
I20260929 01:50:51.547768 5179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1: Bootstrap starting.
I20260929 01:50:51.548323 5037 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:50:51.549396 5186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:50:51.549698 5186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:51.550552 5037 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:50:51.551486 5179 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1: Neither blocks nor log segments found. Creating new log.
I20260929 01:50:51.552683 5179 log.cc:826] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1: Log is configured to *not* fsync() on all Append() calls
I20260929 01:50:51.555215 5179 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1: No bootstrap required, opened a new log
W20260929 01:50:51.558923 4991 tablet.cc:2433] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:50:51.561216 5179 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:50:51.561545 5179 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:50:51.561671 5179 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c786aa789c24cdf89db361dfe2a28e1, State: Initialized, Role: FOLLOWER
I20260929 01:50:51.562471 5179 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:50:51.564181 5200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:50:51.564520 5200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:51.565537 5179 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:50:51.567376 5198 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:50:51.567544 5198 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:50:51.564291 4971 master_runner.cc:187] Error getting master registration for 127.0.71.125:42769: OK, instance_id { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" instance_seqno: 1790646651026207 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:50:51.571004 4971 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:50:51.599617 5211 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:50:51.599952 5211 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:50:51.588593 5110 master_runner.cc:187] Error getting master registration for 127.0.71.125:42769: OK, instance_id { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" instance_seqno: 1790646651026207 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:50:51.601475 5110 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:50:51.728901 5133 tablet.cc:2433] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:50:51.731029 5108 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } attempt: 5
I20260929 01:50:51.744576 5108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64: Bootstrap starting.
I20260929 01:50:51.749073 5108 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64: Neither blocks nor log segments found. Creating new log.
I20260929 01:50:51.750504 5108 log.cc:826] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64: Log is configured to *not* fsync() on all Append() calls
I20260929 01:50:51.753752 5108 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64: No bootstrap required, opened a new log
I20260929 01:50:51.761510 5108 raft_consensus.cc:359] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:50:51.761931 5108 raft_consensus.cc:385] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:50:51.762059 5108 raft_consensus.cc:740] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32119a63667f414c9e4851a7a4a5db64, State: Initialized, Role: FOLLOWER
I20260929 01:50:51.762768 5108 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:50:51.763849 5214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:50:51.764253 5214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:51.765488 5108 sys_catalog.cc:565] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:50:51.781913 5039 master_runner.cc:264] No leader master found from master 32119a63667f414c9e4851a7a4a5db64
I20260929 01:50:51.782055 5039 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:50:51.785104 5225 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:50:51.785295 5225 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:50:51.917204 5186 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:50:51.917527 5186 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:50:51.919601 5186 leader_election.cc:290] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 32119a63667f414c9e4851a7a4a5db64 (127.0.71.125:42769), 6c786aa789c24cdf89db361dfe2a28e1 (127.0.71.124:38453)
I20260929 01:50:51.921073 5083 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32119a63667f414c9e4851a7a4a5db64" is_pre_election: true
I20260929 01:50:51.921639 5083 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea367d8bb04af49f22eaafc8bd4d03 in term 0.
I20260929 01:50:51.922514 4987 leader_election.cc:304] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32119a63667f414c9e4851a7a4a5db64, 4bea367d8bb04af49f22eaafc8bd4d03; no voters:
I20260929 01:50:51.923677 5154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c786aa789c24cdf89db361dfe2a28e1" is_pre_election: true
I20260929 01:50:51.924193 5154 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea367d8bb04af49f22eaafc8bd4d03 in term 0.
I20260929 01:50:51.925266 5186 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:50:51.925434 5186 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:50:51.925555 5186 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:50:51.929445 5186 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:50:51.931185 5083 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "32119a63667f414c9e4851a7a4a5db64"
I20260929 01:50:51.931414 5083 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:50:51.935408 5083 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea367d8bb04af49f22eaafc8bd4d03 in term 1.
I20260929 01:50:51.936720 4987 leader_election.cc:304] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32119a63667f414c9e4851a7a4a5db64, 4bea367d8bb04af49f22eaafc8bd4d03; no voters:
I20260929 01:50:51.940970 5154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "6c786aa789c24cdf89db361dfe2a28e1"
I20260929 01:50:51.941208 5154 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:50:51.945034 5154 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea367d8bb04af49f22eaafc8bd4d03 in term 1.
I20260929 01:50:51.945998 5186 leader_election.cc:290] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [CANDIDATE]: Term 1 election: Requested vote from peers 32119a63667f414c9e4851a7a4a5db64 (127.0.71.125:42769), 6c786aa789c24cdf89db361dfe2a28e1 (127.0.71.124:38453)
I20260929 01:50:51.948316 5186 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:50:51.948975 5186 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 1 LEADER]: Becoming Leader. State: Replica: 4bea367d8bb04af49f22eaafc8bd4d03, State: Running, Role: LEADER
I20260929 01:50:51.949728 5186 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:50:51.963604 5227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bea367d8bb04af49f22eaafc8bd4d03. Latest consensus state: current_term: 1 leader_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:50:51.964092 5227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:51.972047 5229 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:50:51.974989 5229 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:50:51.997524 5154 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [term 1 FOLLOWER]: Refusing update from remote peer 4bea367d8bb04af49f22eaafc8bd4d03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:50:51.998934 5227 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:50:52.007143 5083 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 1 FOLLOWER]: Refusing update from remote peer 4bea367d8bb04af49f22eaafc8bd4d03: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:50:52.008324 5227 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 01:50:52.014681 5062 tablet.cc:2433] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:50:52.033025 5214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bea367d8bb04af49f22eaafc8bd4d03. Latest consensus state: current_term: 1 leader_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:50:52.033304 5214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:52.052871 5230 mvcc.cc:204] Tried to move back new op lower bound from 7334488686567940096 to 7334488686412161024. Current Snapshot: MvccSnapshot[applied={T|T < 7334488686567940096}]
I20260929 01:50:52.054665 5232 mvcc.cc:204] Tried to move back new op lower bound from 7334488686567940096 to 7334488686412161024. Current Snapshot: MvccSnapshot[applied={T|T < 7334488686567940096}]
I20260929 01:50:52.057667 5227 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:50:52.057940 5227 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:52.065019 5214 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:50:52.065249 5214 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:52.067855 5229 catalog_manager.cc:1396] Generated new cluster ID: 8fc1f10b9eb142649bb4c0f1cb2126a6
I20260929 01:50:52.067970 5229 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:50:52.073112 5200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bea367d8bb04af49f22eaafc8bd4d03. Latest consensus state: current_term: 1 leader_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:50:52.073370 5200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:52.074644 5186 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:50:52.074888 5186 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:52.077642 5231 mvcc.cc:204] Tried to move back new op lower bound from 7334488686567940096 to 7334488686412161024. Current Snapshot: MvccSnapshot[applied={T|T < 7334488686567940096}]
I20260929 01:50:52.097488 5200 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:50:52.097765 5200 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:52.125363 5229 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 01:50:52.127347 5229 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:50:52.140134 5229 catalog_manager.cc:6219] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: Generated new TSK 0
I20260929 01:50:52.141342 5229 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:50:52.173306 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--builtin_ntp_servers=127.0.71.84:38621,127.0.71.85:33511,127.0.71.86:38913,127.0.71.87:45783,127.0.71.88:43239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260929 01:50:52.497939 5238 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:52.498271 5238 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:52.498399 5238 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:52.507012 5238 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:52.507239 5238 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:50:52.517683 5238 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38621,127.0.71.85:33511,127.0.71.86:38913,127.0.71.87:45783,127.0.71.88:43239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:52.519670 5238 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:52.521489 5238 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:52.530661 5244 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:52.530813 5243 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:52.531291 5246 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:52.532115 5238 server_base.cc:1061] running on GCE node
I20260929 01:50:52.532719 5238 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:52.533881 5238 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:52.537181 5238 hybrid_clock.cc:648] HybridClock initialized: now 1790646652537016 us; error 131 us; skew 500 ppm
I20260929 01:50:52.537513 5238 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:52.539778 5238 webserver.cc:533] Webserver started at http://127.0.71.65:40363/ using document root <none> and password file <none>
I20260929 01:50:52.540585 5238 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:52.540707 5238 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:52.541061 5238 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:50:52.543370 5238 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "511afaa12e1c4e65a2c8347b29a2e5d1"
format_stamp: "Formatted at 2026-09-29 01:50:52 on dist-test-slave-vz23"
I20260929 01:50:52.544113 5238 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "511afaa12e1c4e65a2c8347b29a2e5d1"
format_stamp: "Formatted at 2026-09-29 01:50:52 on dist-test-slave-vz23"
I20260929 01:50:52.549240 5238 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 01:50:52.552623 5252 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:52.553968 5238 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:50:52.554152 5238 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "511afaa12e1c4e65a2c8347b29a2e5d1"
format_stamp: "Formatted at 2026-09-29 01:50:52 on dist-test-slave-vz23"
I20260929 01:50:52.554363 5238 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:52.571656 5238 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:52.572414 5238 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:52.572685 5238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:52.573551 5238 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:52.575083 5238 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:52.575166 5238 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:52.575242 5238 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:52.575295 5238 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:52.602860 5211 catalog_manager.cc:1308] Loaded cluster ID: 8fc1f10b9eb142649bb4c0f1cb2126a6
I20260929 01:50:52.603010 5211 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1: loading cluster ID for follower catalog manager: success
I20260929 01:50:52.607735 5211 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1: acquiring CA information for follower catalog manager: success
I20260929 01:50:52.610728 5211 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:50:52.612550 5238 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:34683
I20260929 01:50:52.612633 5364 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:34683 every 8 connection(s)
I20260929 01:50:52.614048 5238 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:50:52.621552 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 5238
I20260929 01:50:52.621762 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
I20260929 01:50:52.627576 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.66:0
--local_ip_for_outbound_sockets=127.0.71.66
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--builtin_ntp_servers=127.0.71.84:38621,127.0.71.85:33511,127.0.71.86:38913,127.0.71.87:45783,127.0.71.88:43239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:50:52.645536 5366 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43209
I20260929 01:50:52.645714 5367 heartbeater.cc:344] Connected to a master server at 127.0.71.125:42769
I20260929 01:50:52.645983 5367 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:52.645992 5366 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:52.647112 5365 heartbeater.cc:344] Connected to a master server at 127.0.71.124:38453
I20260929 01:50:52.647133 5367 heartbeater.cc:507] Master 127.0.71.125:42769 requested a full tablet report, sending...
I20260929 01:50:52.647279 5365 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:52.647481 5366 heartbeater.cc:507] Master 127.0.71.126:43209 requested a full tablet report, sending...
I20260929 01:50:52.647791 5365 heartbeater.cc:507] Master 127.0.71.124:38453 requested a full tablet report, sending...
I20260929 01:50:52.649600 5001 ts_manager.cc:194] Registered new tserver with Master: 511afaa12e1c4e65a2c8347b29a2e5d1 (127.0.71.65:34683)
I20260929 01:50:52.649861 5073 ts_manager.cc:194] Registered new tserver with Master: 511afaa12e1c4e65a2c8347b29a2e5d1 (127.0.71.65:34683)
I20260929 01:50:52.649878 5144 ts_manager.cc:194] Registered new tserver with Master: 511afaa12e1c4e65a2c8347b29a2e5d1 (127.0.71.65:34683)
I20260929 01:50:52.651443 5001 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:52591
I20260929 01:50:52.787719 5225 catalog_manager.cc:1308] Loaded cluster ID: 8fc1f10b9eb142649bb4c0f1cb2126a6
I20260929 01:50:52.787854 5225 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64: loading cluster ID for follower catalog manager: success
I20260929 01:50:52.791697 5225 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64: acquiring CA information for follower catalog manager: success
I20260929 01:50:52.794221 5225 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 01:50:52.965735 5375 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:52.966055 5375 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:52.966125 5375 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:52.974845 5375 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:52.975005 5375 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.66
I20260929 01:50:52.985882 5375 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38621,127.0.71.85:33511,127.0.71.86:38913,127.0.71.87:45783,127.0.71.88:43239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:52.987540 5375 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:52.989394 5375 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:52.999272 5381 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:52.999301 5380 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:52.999536 5383 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:53.000715 5375 server_base.cc:1061] running on GCE node
I20260929 01:50:53.001426 5375 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:53.004732 5375 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:53.008112 5375 hybrid_clock.cc:648] HybridClock initialized: now 1790646653007898 us; error 169 us; skew 500 ppm
I20260929 01:50:53.008576 5375 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:53.011564 5375 webserver.cc:533] Webserver started at http://127.0.71.66:35137/ using document root <none> and password file <none>
I20260929 01:50:53.012647 5375 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:53.012810 5375 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:53.013347 5375 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:50:53.016242 5375 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/data/instance:
uuid: "417549d883ef467382ae5014881d55b4"
format_stamp: "Formatted at 2026-09-29 01:50:53 on dist-test-slave-vz23"
I20260929 01:50:53.017272 5375 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/wal/instance:
uuid: "417549d883ef467382ae5014881d55b4"
format_stamp: "Formatted at 2026-09-29 01:50:53 on dist-test-slave-vz23"
I20260929 01:50:53.023520 5375 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 01:50:53.027251 5389 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:53.028934 5375 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:50:53.029120 5375 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/wal
uuid: "417549d883ef467382ae5014881d55b4"
format_stamp: "Formatted at 2026-09-29 01:50:53 on dist-test-slave-vz23"
I20260929 01:50:53.029323 5375 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:53.048987 5375 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:53.049871 5375 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:53.050171 5375 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:53.051111 5375 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:53.052840 5375 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:53.052956 5375 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:53.053083 5375 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:53.053166 5375 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:53.090698 5375 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.66:36355
I20260929 01:50:53.090754 5501 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.66:36355 every 8 connection(s)
I20260929 01:50:53.092103 5375 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
I20260929 01:50:53.094081 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 5375
I20260929 01:50:53.094300 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/wal/instance
I20260929 01:50:53.099195 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.67:0
--local_ip_for_outbound_sockets=127.0.71.67
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--builtin_ntp_servers=127.0.71.84:38621,127.0.71.85:33511,127.0.71.86:38913,127.0.71.87:45783,127.0.71.88:43239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:50:53.112061 5503 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43209
I20260929 01:50:53.112480 5503 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:53.112534 5502 heartbeater.cc:344] Connected to a master server at 127.0.71.124:38453
I20260929 01:50:53.112646 5502 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:53.113725 5502 heartbeater.cc:507] Master 127.0.71.124:38453 requested a full tablet report, sending...
I20260929 01:50:53.113745 5503 heartbeater.cc:507] Master 127.0.71.126:43209 requested a full tablet report, sending...
I20260929 01:50:53.115607 5001 ts_manager.cc:194] Registered new tserver with Master: 417549d883ef467382ae5014881d55b4 (127.0.71.66:36355)
I20260929 01:50:53.115852 5144 ts_manager.cc:194] Registered new tserver with Master: 417549d883ef467382ae5014881d55b4 (127.0.71.66:36355)
I20260929 01:50:53.116679 5001 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.66:36569
I20260929 01:50:53.117177 5504 heartbeater.cc:344] Connected to a master server at 127.0.71.125:42769
I20260929 01:50:53.117372 5504 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:53.117846 5504 heartbeater.cc:507] Master 127.0.71.125:42769 requested a full tablet report, sending...
I20260929 01:50:53.119179 5073 ts_manager.cc:194] Registered new tserver with Master: 417549d883ef467382ae5014881d55b4 (127.0.71.66:36355)
W20260929 01:50:53.439364 5509 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:53.439663 5509 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:53.439734 5509 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:53.448177 5509 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:53.448365 5509 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.67
I20260929 01:50:53.458629 5509 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38621,127.0.71.85:33511,127.0.71.86:38913,127.0.71.87:45783,127.0.71.88:43239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:53.460265 5509 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:53.462190 5509 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:53.472172 5515 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:53.472405 5517 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:53.472141 5514 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:53.473583 5509 server_base.cc:1061] running on GCE node
I20260929 01:50:53.474234 5509 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:53.475337 5509 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:53.476540 5509 hybrid_clock.cc:648] HybridClock initialized: now 1790646653476416 us; error 91 us; skew 500 ppm
I20260929 01:50:53.476917 5509 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:53.479005 5509 webserver.cc:533] Webserver started at http://127.0.71.67:33031/ using document root <none> and password file <none>
I20260929 01:50:53.479745 5509 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:53.479954 5509 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:53.480306 5509 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:50:53.482803 5509 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/data/instance:
uuid: "63980d211a8b43eb930429714233a770"
format_stamp: "Formatted at 2026-09-29 01:50:53 on dist-test-slave-vz23"
I20260929 01:50:53.483572 5509 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/wal/instance:
uuid: "63980d211a8b43eb930429714233a770"
format_stamp: "Formatted at 2026-09-29 01:50:53 on dist-test-slave-vz23"
I20260929 01:50:53.488867 5509 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 01:50:53.492375 5523 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:53.493784 5509 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:50:53.493949 5509 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/wal
uuid: "63980d211a8b43eb930429714233a770"
format_stamp: "Formatted at 2026-09-29 01:50:53 on dist-test-slave-vz23"
I20260929 01:50:53.494131 5509 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:53.512405 5509 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:53.513315 5509 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:53.513592 5509 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:53.514621 5509 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:53.516227 5509 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:53.516343 5509 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:53.516492 5509 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:53.516579 5509 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260929 01:50:53.553045 5509 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.67:46315
I20260929 01:50:53.553139 5635 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.67:46315 every 8 connection(s)
I20260929 01:50:53.554523 5509 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
I20260929 01:50:53.563126 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 5509
I20260929 01:50:53.563367 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/wal/instance
I20260929 01:50:53.578277 5637 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43209
I20260929 01:50:53.578668 5637 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:53.579528 5637 heartbeater.cc:507] Master 127.0.71.126:43209 requested a full tablet report, sending...
I20260929 01:50:53.581143 5001 ts_manager.cc:194] Registered new tserver with Master: 63980d211a8b43eb930429714233a770 (127.0.71.67:46315)
I20260929 01:50:53.582193 5001 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.67:53669
I20260929 01:50:53.582577 5636 heartbeater.cc:344] Connected to a master server at 127.0.71.124:38453
I20260929 01:50:53.582715 5636 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:53.583315 5636 heartbeater.cc:507] Master 127.0.71.124:38453 requested a full tablet report, sending...
I20260929 01:50:53.584465 5144 ts_manager.cc:194] Registered new tserver with Master: 63980d211a8b43eb930429714233a770 (127.0.71.67:46315)
I20260929 01:50:53.586647 5638 heartbeater.cc:344] Connected to a master server at 127.0.71.125:42769
I20260929 01:50:53.586781 5638 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:53.587091 5638 heartbeater.cc:507] Master 127.0.71.125:42769 requested a full tablet report, sending...
I20260929 01:50:53.588104 5073 ts_manager.cc:194] Registered new tserver with Master: 63980d211a8b43eb930429714233a770 (127.0.71.67:46315)
I20260929 01:50:53.592046 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:50:53.600037 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:50:53.600291 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 4971
I20260929 01:50:53.601383 5032 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:53.653890 5366 heartbeater.cc:499] Master 127.0.71.126:43209 was elected leader, sending a full tablet report...
I20260929 01:50:53.709143 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 4971
I20260929 01:50:53.730381 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:43209
--webserver_interface=127.0.71.126
--webserver_port=35505
--builtin_ntp_servers=127.0.71.84:38621,127.0.71.85:33511,127.0.71.86:38913,127.0.71.87:45783,127.0.71.88:43239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453 with env {}
W20260929 01:50:54.087080 5644 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:54.087430 5644 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:54.087553 5644 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:54.095988 5644 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:50:54.096086 5644 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:54.096174 5644 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:50:54.096253 5644 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:50:54.106853 5644 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38621,127.0.71.85:33511,127.0.71.86:38913,127.0.71.87:45783,127.0.71.88:43239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:43209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=35505
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:54.108624 5644 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:54.110445 5644 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:54.119163 5649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:54.119343 5503 heartbeater.cc:499] Master 127.0.71.126:43209 was elected leader, sending a full tablet report...
W20260929 01:50:54.120209 5650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:54.120478 5652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:54.121806 5644 server_base.cc:1061] running on GCE node
I20260929 01:50:54.122550 5644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:54.124099 5644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
W20260929 01:50:54.125041 5503 heartbeater.cc:646] Failed to heartbeat to 127.0.71.126:43209 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.71.126:43209: connect: Connection refused (error 111)
I20260929 01:50:54.127389 5644 hybrid_clock.cc:648] HybridClock initialized: now 1790646654127230 us; error 128 us; skew 500 ppm
I20260929 01:50:54.127760 5644 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:54.130044 5644 webserver.cc:533] Webserver started at http://127.0.71.126:35505/ using document root <none> and password file <none>
I20260929 01:50:54.130779 5644 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:54.130882 5644 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:54.136793 5644 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 01:50:54.139947 5659 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:54.141461 5644 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:50:54.141650 5644 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "4bea367d8bb04af49f22eaafc8bd4d03"
format_stamp: "Formatted at 2026-09-29 01:50:50 on dist-test-slave-vz23"
I20260929 01:50:54.142378 5644 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:54.177695 5644 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:54.178650 5644 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:54.178982 5644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:54.201694 5644 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:43209
I20260929 01:50:54.201716 5710 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:43209 every 8 connection(s)
I20260929 01:50:54.203491 5644 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:50:54.210297 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 5644
I20260929 01:50:54.210608 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 5238
I20260929 01:50:54.211421 5711 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:50:54.215639 5711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: Bootstrap starting.
I20260929 01:50:54.219691 5360 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:54.234001 5711 log.cc:826] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: Log is configured to *not* fsync() on all Append() calls
I20260929 01:50:54.242796 5711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:50:54.243316 5711 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: Bootstrap complete.
I20260929 01:50:54.250226 5711 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:50:54.250954 5711 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea367d8bb04af49f22eaafc8bd4d03, State: Initialized, Role: FOLLOWER
I20260929 01:50:54.251729 5711 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:50:54.252605 5717 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:50:54.252903 5717 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:54.253569 5711 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:50:54.263983 5728 catalog_manager.cc:1308] Loaded cluster ID: 8fc1f10b9eb142649bb4c0f1cb2126a6
I20260929 01:50:54.264209 5728 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: loading cluster ID for follower catalog manager: success
I20260929 01:50:54.270785 5728 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: acquiring CA information for follower catalog manager: success
I20260929 01:50:54.275223 5728 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:50:54.302628 5644 master_runner.cc:264] No leader master found from master 4bea367d8bb04af49f22eaafc8bd4d03
I20260929 01:50:54.302809 5644 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:50:54.338310 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 5238
I20260929 01:50:54.355664 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:34683
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--webserver_port=40363
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:38621,127.0.71.85:33511,127.0.71.86:38913,127.0.71.87:45783,127.0.71.88:43239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:50:54.585567 5637 heartbeater.cc:499] Master 127.0.71.126:43209 was elected leader, sending a full tablet report...
I20260929 01:50:54.606298 5676 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "63980d211a8b43eb930429714233a770" instance_seqno: 1790646653544079) as {username='slave'} at 127.0.71.67:33561; Asking this server to re-register.
I20260929 01:50:54.607586 5637 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:54.609332 5676 ts_manager.cc:194] Registered new tserver with Master: 63980d211a8b43eb930429714233a770 (127.0.71.67:46315)
W20260929 01:50:54.729544 5734 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:54.729871 5734 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:54.730032 5734 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:54.740557 5734 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:54.740759 5734 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:50:54.751233 5734 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38621,127.0.71.85:33511,127.0.71.86:38913,127.0.71.87:45783,127.0.71.88:43239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:34683
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=40363
--tserver_master_addrs=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:54.752959 5734 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:54.754892 5734 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:54.765218 5740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:54.765245 5741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:54.766090 5743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:54.766530 5734 server_base.cc:1061] running on GCE node
I20260929 01:50:54.767194 5734 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:54.768416 5734 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:54.769841 5734 hybrid_clock.cc:648] HybridClock initialized: now 1790646654769715 us; error 90 us; skew 500 ppm
I20260929 01:50:54.770196 5734 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:54.772531 5734 webserver.cc:533] Webserver started at http://127.0.71.65:40363/ using document root <none> and password file <none>
I20260929 01:50:54.773272 5734 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:54.773409 5734 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:54.778895 5734 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:50:54.781886 5749 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:54.783300 5734 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.003s
I20260929 01:50:54.783337 5750 raft_consensus.cc:493] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea367d8bb04af49f22eaafc8bd4d03)
I20260929 01:50:54.783572 5750 raft_consensus.cc:515] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:50:54.783496 5734 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "511afaa12e1c4e65a2c8347b29a2e5d1"
format_stamp: "Formatted at 2026-09-29 01:50:52 on dist-test-slave-vz23"
I20260929 01:50:54.784293 5734 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:54.785039 5750 leader_election.cc:290] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4bea367d8bb04af49f22eaafc8bd4d03 (127.0.71.126:43209), 6c786aa789c24cdf89db361dfe2a28e1 (127.0.71.124:38453)
I20260929 01:50:54.785772 5154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32119a63667f414c9e4851a7a4a5db64" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6c786aa789c24cdf89db361dfe2a28e1" is_pre_election: true
I20260929 01:50:54.793265 5686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32119a63667f414c9e4851a7a4a5db64" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" is_pre_election: true
I20260929 01:50:54.793668 5686 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32119a63667f414c9e4851a7a4a5db64 in term 1.
I20260929 01:50:54.794209 5057 leader_election.cc:304] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32119a63667f414c9e4851a7a4a5db64, 4bea367d8bb04af49f22eaafc8bd4d03; no voters: 6c786aa789c24cdf89db361dfe2a28e1
I20260929 01:50:54.794559 5750 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 01:50:54.794684 5750 raft_consensus.cc:493] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 4bea367d8bb04af49f22eaafc8bd4d03)
I20260929 01:50:54.794870 5750 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:50:54.796958 5750 raft_consensus.cc:515] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:50:54.798014 5750 leader_election.cc:290] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [CANDIDATE]: Term 2 election: Requested vote from peers 4bea367d8bb04af49f22eaafc8bd4d03 (127.0.71.126:43209), 6c786aa789c24cdf89db361dfe2a28e1 (127.0.71.124:38453)
I20260929 01:50:54.798017 5686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32119a63667f414c9e4851a7a4a5db64" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4bea367d8bb04af49f22eaafc8bd4d03"
I20260929 01:50:54.798070 5154 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32119a63667f414c9e4851a7a4a5db64" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "6c786aa789c24cdf89db361dfe2a28e1"
I20260929 01:50:54.798187 5686 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:50:54.803344 5686 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32119a63667f414c9e4851a7a4a5db64 in term 2.
I20260929 01:50:54.803783 5057 leader_election.cc:304] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32119a63667f414c9e4851a7a4a5db64, 4bea367d8bb04af49f22eaafc8bd4d03; no voters: 6c786aa789c24cdf89db361dfe2a28e1
I20260929 01:50:54.804144 5750 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:50:54.804579 5750 raft_consensus.cc:697] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 2 LEADER]: Becoming Leader. State: Replica: 32119a63667f414c9e4851a7a4a5db64, State: Running, Role: LEADER
I20260929 01:50:54.805022 5750 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:50:54.806604 5756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32119a63667f414c9e4851a7a4a5db64. Latest consensus state: current_term: 2 leader_uuid: "32119a63667f414c9e4851a7a4a5db64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:50:54.806850 5756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:54.807495 5758 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:50:54.809621 5734 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:54.809687 5758 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:50:54.810672 5734 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:54.810699 5758 catalog_manager.cc:1308] Loaded cluster ID: 8fc1f10b9eb142649bb4c0f1cb2126a6
I20260929 01:50:54.810791 5758 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:50:54.810995 5734 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:54.812269 5758 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:50:54.812237 5734 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:54.813395 5758 catalog_manager.cc:6230] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64: Loaded TSK: 0
I20260929 01:50:54.814517 5758 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:50:54.814515 5734 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:54.814597 5734 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:54.814733 5734 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:54.814800 5734 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:54.849604 5734 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:34683
I20260929 01:50:54.849675 5866 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:34683 every 8 connection(s)
I20260929 01:50:54.851152 5734 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:50:54.852605 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 5734
I20260929 01:50:54.852890 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 5734
I20260929 01:50:54.895388 5868 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43209
I20260929 01:50:54.895916 5868 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:54.897457 5868 heartbeater.cc:507] Master 127.0.71.126:43209 requested a full tablet report, sending...
I20260929 01:50:54.899963 5676 ts_manager.cc:194] Registered new tserver with Master: 511afaa12e1c4e65a2c8347b29a2e5d1 (127.0.71.65:34683)
I20260929 01:50:54.905997 5867 heartbeater.cc:344] Connected to a master server at 127.0.71.124:38453
I20260929 01:50:54.906203 5867 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:54.907125 5867 heartbeater.cc:507] Master 127.0.71.124:38453 requested a full tablet report, sending...
I20260929 01:50:54.909134 5144 ts_manager.cc:194] Re-registered known tserver with Master: 511afaa12e1c4e65a2c8347b29a2e5d1 (127.0.71.65:34683)
I20260929 01:50:54.910470 5878 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 4bea367d8bb04af49f22eaafc8bd4d03)
I20260929 01:50:54.910688 5878 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:50:54.912191 5862 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:54.913532 5878 leader_election.cc:290] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 4bea367d8bb04af49f22eaafc8bd4d03 (127.0.71.126:43209), 32119a63667f414c9e4851a7a4a5db64 (127.0.71.125:42769)
I20260929 01:50:54.914234 5083 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c786aa789c24cdf89db361dfe2a28e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "32119a63667f414c9e4851a7a4a5db64" is_pre_election: true
I20260929 01:50:54.922536 5686 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c786aa789c24cdf89db361dfe2a28e1" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" is_pre_election: true
I20260929 01:50:54.922945 5686 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6c786aa789c24cdf89db361dfe2a28e1 in current term 2: Already voted for candidate 32119a63667f414c9e4851a7a4a5db64 in this term.
I20260929 01:50:54.923664 5128 leader_election.cc:304] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c786aa789c24cdf89db361dfe2a28e1; no voters: 32119a63667f414c9e4851a7a4a5db64, 4bea367d8bb04af49f22eaafc8bd4d03
I20260929 01:50:54.924149 5878 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:50:54.926777 5878 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:50:55.006596 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 5734
I20260929 01:50:55.025250 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:34683
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--webserver_port=40363
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:38621,127.0.71.85:33511,127.0.71.86:38913,127.0.71.87:45783,127.0.71.88:43239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {}
I20260929 01:50:55.131623 5503 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43209
I20260929 01:50:55.131893 5503 heartbeater.cc:499] Master 127.0.71.126:43209 was elected leader, sending a full tablet report...
I20260929 01:50:55.132833 5676 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "417549d883ef467382ae5014881d55b4" instance_seqno: 1790646653081770) as {username='slave'} at 127.0.71.66:33561; Asking this server to re-register.
I20260929 01:50:55.133388 5503 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:55.134415 5676 ts_manager.cc:194] Registered new tserver with Master: 417549d883ef467382ae5014881d55b4 (127.0.71.66:36355)
I20260929 01:50:55.217286 5686 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 2 FOLLOWER]: Refusing update from remote peer 32119a63667f414c9e4851a7a4a5db64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 01:50:55.218149 5756 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:50:55.221688 5883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32119a63667f414c9e4851a7a4a5db64. Latest consensus state: current_term: 2 leader_uuid: "32119a63667f414c9e4851a7a4a5db64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:50:55.221957 5883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:55.223907 5756 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "32119a63667f414c9e4851a7a4a5db64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:50:55.224170 5756 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:55.224982 5154 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [term 2 FOLLOWER]: Refusing update from remote peer 32119a63667f414c9e4851a7a4a5db64: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 01:50:55.225674 5756 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:50:55.225772 5883 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "32119a63667f414c9e4851a7a4a5db64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:50:55.225934 5883 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:55.233484 5890 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "32119a63667f414c9e4851a7a4a5db64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:50:55.233593 5878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32119a63667f414c9e4851a7a4a5db64. Latest consensus state: current_term: 2 leader_uuid: "32119a63667f414c9e4851a7a4a5db64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:50:55.233857 5878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:55.234027 5890 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:50:55.370368 5881 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:55.370676 5881 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:55.370812 5881 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:50:55.370895 5881 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:55.379197 5881 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:55.379391 5881 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:50:55.389345 5881 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38621,127.0.71.85:33511,127.0.71.86:38913,127.0.71.87:45783,127.0.71.88:43239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:34683
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=40363
--tserver_master_addrs=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:55.390949 5881 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:55.392759 5881 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:55.402350 5896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:55.402734 5895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:55.403076 5898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:55.403736 5881 server_base.cc:1061] running on GCE node
I20260929 01:50:55.404333 5881 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:55.405489 5881 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:55.406848 5881 hybrid_clock.cc:648] HybridClock initialized: now 1790646655406661 us; error 133 us; skew 500 ppm
I20260929 01:50:55.407119 5881 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:55.409286 5881 webserver.cc:533] Webserver started at http://127.0.71.65:40363/ using document root <none> and password file <none>
I20260929 01:50:55.409984 5881 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:55.410084 5881 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:55.415490 5881 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:50:55.418203 5904 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:55.419559 5881 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:50:55.419697 5881 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "511afaa12e1c4e65a2c8347b29a2e5d1"
format_stamp: "Formatted at 2026-09-29 01:50:52 on dist-test-slave-vz23"
I20260929 01:50:55.420342 5881 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:55.434815 5881 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:55.435624 5881 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:55.435853 5881 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:55.436818 5881 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:55.438529 5881 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:55.438612 5881 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:55.438747 5881 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:55.438831 5881 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:50:55.439198 5881 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:50:55.439316 5881 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:50:55.439389 5881 fault_injection.cc:59] injecting fault for kudu (pid 5881): FLAGS_fault_before_start (process will exit)
I20260929 01:50:55.454504 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:34683
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--webserver_port=40363
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:38621,127.0.71.85:33511,127.0.71.86:38913,127.0.71.87:45783,127.0.71.88:43239
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:50:55.720628 5886 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "32119a63667f414c9e4851a7a4a5db64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:50:55.720904 5886 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [sys.catalog]: This master's current role is: LEADER
W20260929 01:50:55.791517 5912 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:55.791815 5912 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:55.791888 5912 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:55.799991 5912 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:55.800148 5912 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:50:55.810022 5912 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38621,127.0.71.85:33511,127.0.71.86:38913,127.0.71.87:45783,127.0.71.88:43239
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:34683
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=40363
--tserver_master_addrs=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:55.811820 5912 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:55.813719 5912 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:55.823966 5917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:55.823992 5918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:55.825111 5920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:55.825541 5912 server_base.cc:1061] running on GCE node
I20260929 01:50:55.826104 5912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:55.827240 5912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:55.828507 5912 hybrid_clock.cc:648] HybridClock initialized: now 1790646655828313 us; error 119 us; skew 500 ppm
I20260929 01:50:55.828819 5912 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:55.830981 5912 webserver.cc:533] Webserver started at http://127.0.71.65:40363/ using document root <none> and password file <none>
I20260929 01:50:55.831709 5912 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:55.831837 5912 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:55.837524 5912 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260929 01:50:55.840219 5926 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:55.841679 5912 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:50:55.841845 5912 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "511afaa12e1c4e65a2c8347b29a2e5d1"
format_stamp: "Formatted at 2026-09-29 01:50:52 on dist-test-slave-vz23"
I20260929 01:50:55.842540 5912 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:55.861593 5912 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:55.862454 5912 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:55.862740 5912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:55.863703 5912 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:55.865413 5912 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:55.865513 5912 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:55.865654 5912 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:55.865722 5912 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:55.899906 5912 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:34683
I20260929 01:50:55.899971 6038 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:34683 every 8 connection(s)
I20260929 01:50:55.901391 5912 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:50:55.902578 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 5912
I20260929 01:50:55.902908 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 5912
I20260929 01:50:55.936267 6041 heartbeater.cc:344] Connected to a master server at 127.0.71.125:42769
I20260929 01:50:55.936694 6040 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43209
I20260929 01:50:55.936761 6041 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:55.936775 6039 heartbeater.cc:344] Connected to a master server at 127.0.71.124:38453
I20260929 01:50:55.936877 6040 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:55.936924 6039 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:55.937929 6034 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:55.938020 6040 heartbeater.cc:507] Master 127.0.71.126:43209 requested a full tablet report, sending...
I20260929 01:50:55.938048 6039 heartbeater.cc:507] Master 127.0.71.124:38453 requested a full tablet report, sending...
I20260929 01:50:55.939975 5144 ts_manager.cc:194] Re-registered known tserver with Master: 511afaa12e1c4e65a2c8347b29a2e5d1 (127.0.71.65:34683)
I20260929 01:50:55.940259 5676 ts_manager.cc:194] Re-registered known tserver with Master: 511afaa12e1c4e65a2c8347b29a2e5d1 (127.0.71.65:34683)
I20260929 01:50:56.026695 6041 heartbeater.cc:507] Master 127.0.71.125:42769 requested a full tablet report, sending...
I20260929 01:50:56.027974 5073 ts_manager.cc:194] Re-registered known tserver with Master: 511afaa12e1c4e65a2c8347b29a2e5d1 (127.0.71.65:34683)
I20260929 01:50:56.028227 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 5912
I20260929 01:50:56.029841 5073 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:47361
W20260929 01:50:56.050239 5057 connection.cc:573] server connection from 127.0.71.65:47361 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:50:56.050302 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 5375
I20260929 01:50:56.058593 5497 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:56.148664 5504 heartbeater.cc:499] Master 127.0.71.125:42769 was elected leader, sending a full tablet report...
I20260929 01:50:56.149386 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 5375
W20260929 01:50:56.167285 5060 connection.cc:573] server connection from 127.0.71.66:50291 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 01:50:56.167687 5661 connection.cc:573] server connection from 127.0.71.66:33561 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:50:56.168018 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 5509
I20260929 01:50:56.176188 5631 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:56.267098 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 5509
I20260929 01:50:56.283406 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 5644
I20260929 01:50:56.289863 5706 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:56.395537 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 5644
I20260929 01:50:56.415607 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 5039
I20260929 01:50:56.416855 5103 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:56.527982 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 5039
I20260929 01:50:56.547855 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 5110
I20260929 01:50:56.548977 5174 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:50:56.654592 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 5110
2026-09-29T01:50:56Z chronyd exiting
2026-09-29T01:50:56Z chronyd exiting
2026-09-29T01:50:56Z chronyd exiting
2026-09-29T01:50:56Z chronyd exiting
2026-09-29T01:50:56Z chronyd exiting
2026-09-29T01:50:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:50:56Z Disabled control of system clock
2026-09-29T01:50:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:50:56Z Disabled control of system clock
2026-09-29T01:50:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:50:56Z Disabled control of system clock
2026-09-29T01:50:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:50:56Z Disabled control of system clock
2026-09-29T01:50:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:50:56Z Disabled control of system clock
I20260929 01:50:56.833037 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:43209
--webserver_interface=127.0.71.126
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:35355,127.0.71.85:34745,127.0.71.86:38995,127.0.71.87:44969,127.0.71.88:39357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453 with env {}
W20260929 01:50:57.179819 6071 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:57.180184 6071 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:57.180338 6071 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:57.188760 6071 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:50:57.188858 6071 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:57.188952 6071 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:50:57.189031 6071 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:50:57.199558 6071 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35355,127.0.71.85:34745,127.0.71.86:38995,127.0.71.87:44969,127.0.71.88:39357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:43209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:57.201526 6071 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:57.203318 6071 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:57.212040 6077 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:57.212033 6076 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:57.212303 6079 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:57.212940 6071 server_base.cc:1061] running on GCE node
I20260929 01:50:57.213696 6071 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:57.215178 6071 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:57.216393 6071 hybrid_clock.cc:648] HybridClock initialized: now 1790646657216263 us; error 99 us; skew 500 ppm
I20260929 01:50:57.216780 6071 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:57.219223 6071 webserver.cc:533] Webserver started at http://127.0.71.126:36119/ using document root <none> and password file <none>
I20260929 01:50:57.219951 6071 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:57.220044 6071 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:57.225340 6071 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:50:57.228192 6085 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:57.229678 6071 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:50:57.229863 6071 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "4bea367d8bb04af49f22eaafc8bd4d03"
format_stamp: "Formatted at 2026-09-29 01:50:50 on dist-test-slave-vz23"
I20260929 01:50:57.230506 6071 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:57.261492 6071 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:57.262385 6071 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:57.262692 6071 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:57.282788 6071 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:43209
I20260929 01:50:57.282809 6136 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:43209 every 8 connection(s)
I20260929 01:50:57.284408 6071 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:50:57.290330 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 6071
I20260929 01:50:57.290521 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:50:57.291729 6137 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:50:57.291705 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:42769
--webserver_interface=127.0.71.125
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:35355,127.0.71.85:34745,127.0.71.86:38995,127.0.71.87:44969,127.0.71.88:39357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453 with env {}
I20260929 01:50:57.296990 6137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: Bootstrap starting.
I20260929 01:50:57.313818 6137 log.cc:826] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: Log is configured to *not* fsync() on all Append() calls
I20260929 01:50:57.322386 6137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:50:57.322903 6137 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: Bootstrap complete.
I20260929 01:50:57.329566 6137 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:50:57.330281 6137 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea367d8bb04af49f22eaafc8bd4d03, State: Initialized, Role: FOLLOWER
I20260929 01:50:57.331020 6137 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:50:57.331892 6141 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:50:57.332214 6141 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:57.332805 6137 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:50:57.343761 6152 catalog_manager.cc:1308] Loaded cluster ID: 8fc1f10b9eb142649bb4c0f1cb2126a6
I20260929 01:50:57.345268 6152 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: loading cluster ID for follower catalog manager: success
I20260929 01:50:57.352778 6152 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: acquiring CA information for follower catalog manager: success
I20260929 01:50:57.359007 6152 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:50:57.370786 6071 master_runner.cc:187] Error getting master registration for 127.0.71.125:42769: Network error: Client connection negotiation failed: client connection to 127.0.71.125:42769: connect: Connection refused (error 111),
I20260929 01:50:57.376549 6071 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:50:57.637840 6139 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:57.638195 6139 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:57.638314 6139 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:57.646763 6139 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:50:57.646883 6139 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:57.647004 6139 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:50:57.647080 6139 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:50:57.657280 6139 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35355,127.0.71.85:34745,127.0.71.86:38995,127.0.71.87:44969,127.0.71.88:39357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:42769
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:57.658936 6139 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:57.660910 6139 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:57.668857 6160 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:57.668910 6161 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:57.670499 6163 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:57.670903 6139 server_base.cc:1061] running on GCE node
I20260929 01:50:57.671741 6139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:57.673645 6139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:57.674875 6139 hybrid_clock.cc:648] HybridClock initialized: now 1790646657674711 us; error 130 us; skew 500 ppm
I20260929 01:50:57.675279 6139 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:57.677759 6139 webserver.cc:533] Webserver started at http://127.0.71.125:42885/ using document root <none> and password file <none>
I20260929 01:50:57.678632 6139 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:57.678758 6139 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:57.685832 6139 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:50:57.689245 6169 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:57.690940 6139 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:50:57.691118 6139 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "32119a63667f414c9e4851a7a4a5db64"
format_stamp: "Formatted at 2026-09-29 01:50:50 on dist-test-slave-vz23"
I20260929 01:50:57.692036 6139 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:57.709755 6139 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:57.710739 6139 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:57.711025 6139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:57.734458 6139 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:42769
I20260929 01:50:57.734472 6220 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:42769 every 8 connection(s)
I20260929 01:50:57.736151 6139 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:50:57.742089 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 6139
I20260929 01:50:57.742292 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:50:57.743585 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:38453
--webserver_interface=127.0.71.124
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:35355,127.0.71.85:34745,127.0.71.86:38995,127.0.71.87:44969,127.0.71.88:39357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453 with env {}
I20260929 01:50:57.744510 6221 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:50:57.748250 6221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64: Bootstrap starting.
I20260929 01:50:57.764549 6221 log.cc:826] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64: Log is configured to *not* fsync() on all Append() calls
I20260929 01:50:57.773176 6221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:50:57.773687 6221 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64: Bootstrap complete.
I20260929 01:50:57.780129 6221 raft_consensus.cc:359] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:50:57.780889 6221 raft_consensus.cc:740] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32119a63667f414c9e4851a7a4a5db64, State: Initialized, Role: FOLLOWER
I20260929 01:50:57.781606 6221 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:50:57.782461 6225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:50:57.782773 6225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:57.783474 6221 sys_catalog.cc:565] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:50:57.793552 6236 catalog_manager.cc:1308] Loaded cluster ID: 8fc1f10b9eb142649bb4c0f1cb2126a6
I20260929 01:50:57.793736 6236 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64: loading cluster ID for follower catalog manager: success
I20260929 01:50:57.797864 6236 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64: acquiring CA information for follower catalog manager: success
I20260929 01:50:57.800712 6236 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:50:57.827561 6139 master_runner.cc:187] Error getting master registration for 127.0.71.124:38453: Network error: Client connection negotiation failed: client connection to 127.0.71.124:38453: connect: Connection refused (error 111),
I20260929 01:50:57.833294 6139 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:50:58.082576 6223 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:58.082939 6223 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:58.083060 6223 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:58.091547 6223 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:50:58.091671 6223 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:58.091782 6223 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:50:58.091859 6223 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:50:58.102157 6223 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35355,127.0.71.85:34745,127.0.71.86:38995,127.0.71.87:44969,127.0.71.88:39357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:38453
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:58.103811 6223 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:58.105638 6223 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:58.113595 6245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:58.113782 6244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:58.115338 6247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:58.115932 6223 server_base.cc:1061] running on GCE node
I20260929 01:50:58.116904 6223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:58.118575 6223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:58.119812 6223 hybrid_clock.cc:648] HybridClock initialized: now 1790646658119667 us; error 116 us; skew 500 ppm
I20260929 01:50:58.120182 6223 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:58.122553 6223 webserver.cc:533] Webserver started at http://127.0.71.124:39035/ using document root <none> and password file <none>
I20260929 01:50:58.123299 6223 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:58.123421 6223 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:58.129140 6223 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 01:50:58.132097 6253 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:58.133672 6223 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:50:58.133878 6223 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "6c786aa789c24cdf89db361dfe2a28e1"
format_stamp: "Formatted at 2026-09-29 01:50:51 on dist-test-slave-vz23"
I20260929 01:50:58.134610 6223 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:58.147891 6223 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:58.148751 6223 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:58.149036 6223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:58.170557 6223 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:38453
I20260929 01:50:58.170557 6304 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:38453 every 8 connection(s)
I20260929 01:50:58.172202 6223 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:50:58.179138 6305 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:50:58.181965 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 6223
I20260929 01:50:58.182143 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:50:58.182463 6305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1: Bootstrap starting.
I20260929 01:50:58.202992 6305 log.cc:826] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1: Log is configured to *not* fsync() on all Append() calls
I20260929 01:50:58.212754 6305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:50:58.213327 6305 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1: Bootstrap complete.
I20260929 01:50:58.219652 6305 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:50:58.220362 6305 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6c786aa789c24cdf89db361dfe2a28e1, State: Initialized, Role: FOLLOWER
I20260929 01:50:58.221042 6305 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:50:58.221800 6310 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:50:58.222041 6310 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:58.222828 6305 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:50:58.233242 6321 catalog_manager.cc:1308] Loaded cluster ID: 8fc1f10b9eb142649bb4c0f1cb2126a6
I20260929 01:50:58.233349 6321 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1: loading cluster ID for follower catalog manager: success
I20260929 01:50:58.237546 6321 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1: acquiring CA information for follower catalog manager: success
I20260929 01:50:58.240116 6321 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:50:58.264134 6223 master_runner.cc:264] No leader master found from master 6c786aa789c24cdf89db361dfe2a28e1
I20260929 01:50:58.264273 6223 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:50:58.636942 6324 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:50:58.637231 6324 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:50:58.639166 6324 leader_election.cc:290] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 32119a63667f414c9e4851a7a4a5db64 (127.0.71.125:42769), 6c786aa789c24cdf89db361dfe2a28e1 (127.0.71.124:38453)
I20260929 01:50:58.648310 6196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32119a63667f414c9e4851a7a4a5db64" is_pre_election: true
I20260929 01:50:58.648506 6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6c786aa789c24cdf89db361dfe2a28e1" is_pre_election: true
I20260929 01:50:58.648954 6196 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea367d8bb04af49f22eaafc8bd4d03 in term 2.
I20260929 01:50:58.649068 6280 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4bea367d8bb04af49f22eaafc8bd4d03 in term 2.
I20260929 01:50:58.649744 6087 leader_election.cc:304] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32119a63667f414c9e4851a7a4a5db64, 4bea367d8bb04af49f22eaafc8bd4d03; no voters:
I20260929 01:50:58.650198 6324 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 01:50:58.650375 6324 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:50:58.650493 6324 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:50:58.654453 6324 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:50:58.655198 6324 leader_election.cc:290] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [CANDIDATE]: Term 3 election: Requested vote from peers 32119a63667f414c9e4851a7a4a5db64 (127.0.71.125:42769), 6c786aa789c24cdf89db361dfe2a28e1 (127.0.71.124:38453)
I20260929 01:50:58.655634 6196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "32119a63667f414c9e4851a7a4a5db64"
I20260929 01:50:58.655790 6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "6c786aa789c24cdf89db361dfe2a28e1"
I20260929 01:50:58.655859 6196 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:50:58.656008 6280 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:50:58.659451 6196 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea367d8bb04af49f22eaafc8bd4d03 in term 3.
I20260929 01:50:58.659904 6280 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4bea367d8bb04af49f22eaafc8bd4d03 in term 3.
I20260929 01:50:58.660024 6087 leader_election.cc:304] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 32119a63667f414c9e4851a7a4a5db64, 4bea367d8bb04af49f22eaafc8bd4d03; no voters:
I20260929 01:50:58.660394 6324 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 01:50:58.660925 6324 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 3 LEADER]: Becoming Leader. State: Replica: 4bea367d8bb04af49f22eaafc8bd4d03, State: Running, Role: LEADER
I20260929 01:50:58.661445 6324 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:50:58.664140 6327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bea367d8bb04af49f22eaafc8bd4d03. Latest consensus state: current_term: 3 leader_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:50:58.664391 6327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:58.665091 6329 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:50:58.667047 6329 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:50:58.667819 6329 catalog_manager.cc:1308] Loaded cluster ID: 8fc1f10b9eb142649bb4c0f1cb2126a6
I20260929 01:50:58.667915 6329 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:50:58.669070 6329 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:50:58.669839 6329 catalog_manager.cc:6230] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: Loaded TSK: 0
I20260929 01:50:58.670672 6329 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:50:58.681587 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--builtin_ntp_servers=127.0.71.84:35355,127.0.71.85:34745,127.0.71.86:38995,127.0.71.87:44969,127.0.71.88:39357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260929 01:50:59.021873 6330 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:59.022213 6330 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:59.022358 6330 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:59.030845 6330 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:59.031071 6330 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:50:59.041762 6330 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35355,127.0.71.85:34745,127.0.71.86:38995,127.0.71.87:44969,127.0.71.88:39357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:59.042315 6280 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [term 3 FOLLOWER]: Refusing update from remote peer 4bea367d8bb04af49f22eaafc8bd4d03: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 01:50:59.043275 6327 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:50:59.043768 6330 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:59.045877 6330 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260929 01:50:59.047454 6334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bea367d8bb04af49f22eaafc8bd4d03. Latest consensus state: current_term: 3 leader_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:50:59.047659 6334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:59.049489 6334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:50:59.049717 6334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:50:59.056877 6340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:59.056984 6339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:59.058188 6342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:59.059505 6327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:50:59.059769 6327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:59.059744 6330 server_base.cc:1061] running on GCE node
I20260929 01:50:59.059804 6196 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 3 FOLLOWER]: Refusing update from remote peer 4bea367d8bb04af49f22eaafc8bd4d03: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 01:50:59.060499 6330 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:59.060731 6344 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [LEADER]: Connected to new peer: Peer: permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:50:59.061899 6330 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:59.064591 6349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:50:59.065001 6349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:59.065191 6330 hybrid_clock.cc:648] HybridClock initialized: now 1790646659064990 us; error 171 us; skew 500 ppm
I20260929 01:50:59.065524 6330 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:59.065487 6346 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4bea367d8bb04af49f22eaafc8bd4d03. Latest consensus state: current_term: 3 leader_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:50:59.065717 6346 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:50:59.067822 6330 webserver.cc:533] Webserver started at http://127.0.71.65:43531/ using document root <none> and password file <none>
I20260929 01:50:59.068672 6330 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:59.068810 6330 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:59.074838 6330 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:50:59.077803 6353 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:59.079308 6330 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:50:59.079612 6330 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "511afaa12e1c4e65a2c8347b29a2e5d1"
format_stamp: "Formatted at 2026-09-29 01:50:52 on dist-test-slave-vz23"
I20260929 01:50:59.080511 6330 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:59.101289 6330 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:59.102186 6330 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:59.102501 6330 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:59.103508 6330 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:59.105391 6330 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:59.105509 6330 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:59.105669 6330 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:59.105753 6330 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:59.142179 6330 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:44677
I20260929 01:50:59.142246 6465 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:44677 every 8 connection(s)
I20260929 01:50:59.143726 6330 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:50:59.151710 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 6330
I20260929 01:50:59.151954 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
I20260929 01:50:59.184473 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.66:0
--local_ip_for_outbound_sockets=127.0.71.66
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--builtin_ntp_servers=127.0.71.84:35355,127.0.71.85:34745,127.0.71.86:38995,127.0.71.87:44969,127.0.71.88:39357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:50:59.193419 6468 heartbeater.cc:344] Connected to a master server at 127.0.71.125:42769
I20260929 01:50:59.193466 6467 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43209
I20260929 01:50:59.193819 6467 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:59.194929 6467 heartbeater.cc:507] Master 127.0.71.126:43209 requested a full tablet report, sending...
I20260929 01:50:59.195272 6466 heartbeater.cc:344] Connected to a master server at 127.0.71.124:38453
I20260929 01:50:59.193816 6468 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:59.195423 6466 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:59.196050 6468 heartbeater.cc:507] Master 127.0.71.125:42769 requested a full tablet report, sending...
I20260929 01:50:59.196146 6466 heartbeater.cc:507] Master 127.0.71.124:38453 requested a full tablet report, sending...
I20260929 01:50:59.197448 6102 ts_manager.cc:194] Registered new tserver with Master: 511afaa12e1c4e65a2c8347b29a2e5d1 (127.0.71.65:44677)
I20260929 01:50:59.197921 6270 ts_manager.cc:194] Registered new tserver with Master: 511afaa12e1c4e65a2c8347b29a2e5d1 (127.0.71.65:44677)
I20260929 01:50:59.198745 6186 ts_manager.cc:194] Registered new tserver with Master: 511afaa12e1c4e65a2c8347b29a2e5d1 (127.0.71.65:44677)
I20260929 01:50:59.199612 6102 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:52283
W20260929 01:50:59.525785 6476 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:59.526085 6476 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:59.526239 6476 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:59.534684 6476 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:50:59.534893 6476 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.66
I20260929 01:50:59.545248 6476 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35355,127.0.71.85:34745,127.0.71.86:38995,127.0.71.87:44969,127.0.71.88:39357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:50:59.547030 6476 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:50:59.548839 6476 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:50:59.558475 6482 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:59.558543 6481 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:50:59.558627 6484 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:50:59.559795 6476 server_base.cc:1061] running on GCE node
I20260929 01:50:59.560518 6476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:50:59.561609 6476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:50:59.562832 6476 hybrid_clock.cc:648] HybridClock initialized: now 1790646659562710 us; error 91 us; skew 500 ppm
I20260929 01:50:59.563165 6476 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:50:59.565443 6476 webserver.cc:533] Webserver started at http://127.0.71.66:33363/ using document root <none> and password file <none>
I20260929 01:50:59.566196 6476 fs_manager.cc:362] Metadata directory not provided
I20260929 01:50:59.566332 6476 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:50:59.571619 6476 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:50:59.574267 6490 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:59.575699 6476 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:50:59.575857 6476 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/wal
uuid: "417549d883ef467382ae5014881d55b4"
format_stamp: "Formatted at 2026-09-29 01:50:53 on dist-test-slave-vz23"
I20260929 01:50:59.576586 6476 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:50:59.579811 6491 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:50:59.580056 6491 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: This master's current role is: LEADER
I20260929 01:50:59.594859 6476 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:50:59.595661 6476 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:50:59.595896 6476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:50:59.596809 6476 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:50:59.598438 6476 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:50:59.598517 6476 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:59.598596 6476 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:50:59.598639 6476 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:50:59.634277 6476 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.66:34601
I20260929 01:50:59.634384 6606 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.66:34601 every 8 connection(s)
I20260929 01:50:59.635910 6476 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
I20260929 01:50:59.640937 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 6476
I20260929 01:50:59.641126 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-1/wal/instance
I20260929 01:50:59.645826 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.67:0
--local_ip_for_outbound_sockets=127.0.71.67
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--builtin_ntp_servers=127.0.71.84:35355,127.0.71.85:34745,127.0.71.86:38995,127.0.71.87:44969,127.0.71.88:39357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:50:59.654188 6607 heartbeater.cc:344] Connected to a master server at 127.0.71.124:38453
I20260929 01:50:59.654934 6607 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:59.655514 6609 heartbeater.cc:344] Connected to a master server at 127.0.71.125:42769
I20260929 01:50:59.655649 6609 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:59.656122 6607 heartbeater.cc:507] Master 127.0.71.124:38453 requested a full tablet report, sending...
I20260929 01:50:59.656342 6609 heartbeater.cc:507] Master 127.0.71.125:42769 requested a full tablet report, sending...
I20260929 01:50:59.657604 6608 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43209
I20260929 01:50:59.657713 6608 heartbeater.cc:461] Registering TS with master...
I20260929 01:50:59.657984 6270 ts_manager.cc:194] Registered new tserver with Master: 417549d883ef467382ae5014881d55b4 (127.0.71.66:34601)
I20260929 01:50:59.658061 6186 ts_manager.cc:194] Registered new tserver with Master: 417549d883ef467382ae5014881d55b4 (127.0.71.66:34601)
I20260929 01:50:59.658143 6608 heartbeater.cc:507] Master 127.0.71.126:43209 requested a full tablet report, sending...
I20260929 01:50:59.659169 6102 ts_manager.cc:194] Registered new tserver with Master: 417549d883ef467382ae5014881d55b4 (127.0.71.66:34601)
I20260929 01:50:59.660005 6102 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.66:43417
W20260929 01:50:59.990185 6614 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:50:59.990595 6614 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:50:59.990762 6614 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:50:59.999797 6614 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:51:00.000034 6614 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.67
I20260929 01:51:00.010673 6614 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35355,127.0.71.85:34745,127.0.71.86:38995,127.0.71.87:44969,127.0.71.88:39357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:51:00.012296 6614 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:51:00.014137 6614 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:51:00.023993 6619 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:00.024076 6620 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:00.024598 6622 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:00.025521 6614 server_base.cc:1061] running on GCE node
I20260929 01:51:00.026201 6614 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:51:00.027405 6614 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:51:00.028770 6614 hybrid_clock.cc:648] HybridClock initialized: now 1790646660028632 us; error 109 us; skew 500 ppm
I20260929 01:51:00.029134 6614 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:51:00.031260 6614 webserver.cc:533] Webserver started at http://127.0.71.67:43435/ using document root <none> and password file <none>
I20260929 01:51:00.031984 6614 fs_manager.cc:362] Metadata directory not provided
I20260929 01:51:00.032122 6614 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:51:00.038005 6614 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:51:00.041031 6628 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:00.042519 6614 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:51:00.042804 6614 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/wal
uuid: "63980d211a8b43eb930429714233a770"
format_stamp: "Formatted at 2026-09-29 01:50:53 on dist-test-slave-vz23"
I20260929 01:51:00.043503 6614 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:51:00.065667 6614 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:51:00.066547 6614 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:51:00.066836 6614 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:51:00.067760 6614 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:51:00.069597 6614 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:51:00.069695 6614 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:00.069833 6614 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:51:00.069900 6614 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:00.106585 6614 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.67:37415
I20260929 01:51:00.106637 6740 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.67:37415 every 8 connection(s)
I20260929 01:51:00.108160 6614 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
I20260929 01:51:00.110237 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 6614
I20260929 01:51:00.110447 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-2/wal/instance
I20260929 01:51:00.134316 6743 heartbeater.cc:344] Connected to a master server at 127.0.71.125:42769
I20260929 01:51:00.134693 6743 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:00.135803 6743 heartbeater.cc:507] Master 127.0.71.125:42769 requested a full tablet report, sending...
I20260929 01:51:00.136932 6741 heartbeater.cc:344] Connected to a master server at 127.0.71.124:38453
I20260929 01:51:00.137054 6741 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:00.137422 6742 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43209
I20260929 01:51:00.137567 6741 heartbeater.cc:507] Master 127.0.71.124:38453 requested a full tablet report, sending...
I20260929 01:51:00.137581 6742 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:00.137681 6186 ts_manager.cc:194] Registered new tserver with Master: 63980d211a8b43eb930429714233a770 (127.0.71.67:37415)
I20260929 01:51:00.138113 6742 heartbeater.cc:507] Master 127.0.71.126:43209 requested a full tablet report, sending...
I20260929 01:51:00.138588 6270 ts_manager.cc:194] Registered new tserver with Master: 63980d211a8b43eb930429714233a770 (127.0.71.67:37415)
I20260929 01:51:00.139122 6102 ts_manager.cc:194] Registered new tserver with Master: 63980d211a8b43eb930429714233a770 (127.0.71.67:37415)
I20260929 01:51:00.140010 6102 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.67:47411
I20260929 01:51:00.153525 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:51:00.160801 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:51:00.160967 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 6071
I20260929 01:51:00.161723 6132 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:51:00.201926 6467 heartbeater.cc:499] Master 127.0.71.126:43209 was elected leader, sending a full tablet report...
I20260929 01:51:00.273311 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 6071
I20260929 01:51:00.295044 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:43209
--webserver_interface=127.0.71.126
--webserver_port=36119
--builtin_ntp_servers=127.0.71.84:35355,127.0.71.85:34745,127.0.71.86:38995,127.0.71.87:44969,127.0.71.88:39357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453 with env {}
W20260929 01:51:00.629102 6749 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:51:00.629436 6749 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:51:00.629530 6749 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:51:00.638444 6749 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:51:00.638545 6749 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:51:00.638648 6749 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:51:00.638738 6749 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:51:00.649662 6749 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35355,127.0.71.85:34745,127.0.71.86:38995,127.0.71.87:44969,127.0.71.88:39357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:43209
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=36119
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:51:00.651305 6749 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:51:00.653218 6749 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:51:00.661520 6754 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:00.662739 6608 heartbeater.cc:499] Master 127.0.71.126:43209 was elected leader, sending a full tablet report...
W20260929 01:51:00.662808 6755 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:00.663663 6749 server_base.cc:1061] running on GCE node
W20260929 01:51:00.663726 6757 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:00.664433 6608 heartbeater.cc:646] Failed to heartbeat to 127.0.71.126:43209 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.71.126:43209: connect: Connection refused (error 111)
I20260929 01:51:00.664894 6749 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:51:00.666548 6749 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:51:00.667905 6749 hybrid_clock.cc:648] HybridClock initialized: now 1790646660667745 us; error 127 us; skew 500 ppm
I20260929 01:51:00.668278 6749 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:51:00.670586 6749 webserver.cc:533] Webserver started at http://127.0.71.126:36119/ using document root <none> and password file <none>
I20260929 01:51:00.671326 6749 fs_manager.cc:362] Metadata directory not provided
I20260929 01:51:00.671463 6749 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:51:00.676991 6749 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.002s
I20260929 01:51:00.680011 6764 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:00.681555 6749 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:51:00.681746 6749 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "4bea367d8bb04af49f22eaafc8bd4d03"
format_stamp: "Formatted at 2026-09-29 01:50:50 on dist-test-slave-vz23"
I20260929 01:51:00.682547 6749 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:51:00.730187 6749 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:51:00.731142 6749 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:51:00.731478 6749 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:51:00.754436 6749 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:43209
I20260929 01:51:00.754513 6815 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:43209 every 8 connection(s)
I20260929 01:51:00.756675 6749 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:51:00.763932 6816 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:51:00.764169 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 6749
I20260929 01:51:00.764524 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 6330
I20260929 01:51:00.768086 6816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: Bootstrap starting.
I20260929 01:51:00.774086 6461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:51:00.787055 6816 log.cc:826] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: Log is configured to *not* fsync() on all Append() calls
I20260929 01:51:00.795956 6816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:51:00.796535 6816 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: Bootstrap complete.
I20260929 01:51:00.803248 6816 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:51:00.803948 6816 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4bea367d8bb04af49f22eaafc8bd4d03, State: Initialized, Role: FOLLOWER
I20260929 01:51:00.804747 6816 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:51:00.805598 6822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:51:00.805896 6822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:00.806722 6816 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:51:00.817294 6833 catalog_manager.cc:1308] Loaded cluster ID: 8fc1f10b9eb142649bb4c0f1cb2126a6
I20260929 01:51:00.817426 6833 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: loading cluster ID for follower catalog manager: success
I20260929 01:51:00.823738 6833 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: acquiring CA information for follower catalog manager: success
I20260929 01:51:00.827188 6833 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:51:00.854414 6749 master_runner.cc:264] No leader master found from master 4bea367d8bb04af49f22eaafc8bd4d03
I20260929 01:51:00.854583 6749 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:51:00.894793 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 6330
I20260929 01:51:00.911839 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:44677
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--webserver_port=43531
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:35355,127.0.71.85:34745,127.0.71.86:38995,127.0.71.87:44969,127.0.71.88:39357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:51:01.142369 6742 heartbeater.cc:499] Master 127.0.71.126:43209 was elected leader, sending a full tablet report...
I20260929 01:51:01.153501 6781 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "63980d211a8b43eb930429714233a770" instance_seqno: 1790646660098286) as {username='slave'} at 127.0.71.67:44457; Asking this server to re-register.
I20260929 01:51:01.154524 6742 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:01.156721 6781 ts_manager.cc:194] Registered new tserver with Master: 63980d211a8b43eb930429714233a770 (127.0.71.67:37415)
W20260929 01:51:01.269490 6839 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:51:01.269817 6839 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:51:01.269987 6839 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:51:01.280117 6839 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:51:01.280357 6839 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:51:01.291926 6839 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35355,127.0.71.85:34745,127.0.71.86:38995,127.0.71.87:44969,127.0.71.88:39357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:44677
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=43531
--tserver_master_addrs=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:51:01.293588 6839 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:51:01.295377 6839 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:51:01.305641 6845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:01.305565 6848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:01.305913 6846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:01.306666 6839 server_base.cc:1061] running on GCE node
I20260929 01:51:01.307242 6839 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:51:01.308487 6839 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:51:01.309793 6839 hybrid_clock.cc:648] HybridClock initialized: now 1790646661309634 us; error 123 us; skew 500 ppm
I20260929 01:51:01.310189 6839 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:51:01.312431 6839 webserver.cc:533] Webserver started at http://127.0.71.65:43531/ using document root <none> and password file <none>
I20260929 01:51:01.313251 6839 fs_manager.cc:362] Metadata directory not provided
I20260929 01:51:01.313390 6839 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:51:01.318815 6839 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.000s sys 0.005s
I20260929 01:51:01.321861 6854 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:01.323342 6839 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 01:51:01.323516 6839 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "511afaa12e1c4e65a2c8347b29a2e5d1"
format_stamp: "Formatted at 2026-09-29 01:50:52 on dist-test-slave-vz23"
I20260929 01:51:01.324213 6839 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:51:01.346601 6839 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:51:01.347468 6839 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:51:01.347765 6839 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:51:01.348812 6839 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:51:01.350665 6839 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:51:01.350737 6839 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:01.350858 6839 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:51:01.350920 6839 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:01.383775 6839 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:44677
I20260929 01:51:01.383819 6966 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:44677 every 8 connection(s)
I20260929 01:51:01.385337 6839 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:51:01.393041 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 6839
I20260929 01:51:01.393330 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 6839
I20260929 01:51:01.411212 6968 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43209
I20260929 01:51:01.411276 6967 heartbeater.cc:344] Connected to a master server at 127.0.71.124:38453
I20260929 01:51:01.411581 6969 heartbeater.cc:344] Connected to a master server at 127.0.71.125:42769
I20260929 01:51:01.411679 6968 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:01.411729 6969 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:01.411686 6967 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:01.411970 6962 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:51:01.499778 6968 heartbeater.cc:507] Master 127.0.71.126:43209 requested a full tablet report, sending...
I20260929 01:51:01.499773 6969 heartbeater.cc:507] Master 127.0.71.125:42769 requested a full tablet report, sending...
I20260929 01:51:01.500278 6967 heartbeater.cc:507] Master 127.0.71.124:38453 requested a full tablet report, sending...
I20260929 01:51:01.501139 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 6839
I20260929 01:51:01.502146 6781 ts_manager.cc:194] Registered new tserver with Master: 511afaa12e1c4e65a2c8347b29a2e5d1 (127.0.71.65:44677)
I20260929 01:51:01.502218 6270 ts_manager.cc:194] Re-registered known tserver with Master: 511afaa12e1c4e65a2c8347b29a2e5d1 (127.0.71.65:44677)
I20260929 01:51:01.503144 6186 ts_manager.cc:194] Re-registered known tserver with Master: 511afaa12e1c4e65a2c8347b29a2e5d1 (127.0.71.65:44677)
I20260929 01:51:01.509265 6979 raft_consensus.cc:493] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4bea367d8bb04af49f22eaafc8bd4d03)
I20260929 01:51:01.509449 6979 raft_consensus.cc:515] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:51:01.511273 6979 leader_election.cc:290] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4bea367d8bb04af49f22eaafc8bd4d03 (127.0.71.126:43209), 6c786aa789c24cdf89db361dfe2a28e1 (127.0.71.124:38453)
I20260929 01:51:01.518831 6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32119a63667f414c9e4851a7a4a5db64" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6c786aa789c24cdf89db361dfe2a28e1" is_pre_election: true
I20260929 01:51:01.519057 6791 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32119a63667f414c9e4851a7a4a5db64" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" is_pre_election: true
I20260929 01:51:01.519636 6791 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 32119a63667f414c9e4851a7a4a5db64 in term 3.
W20260929 01:51:01.520200 6255 connection.cc:573] server connection from 127.0.71.65:52197 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 01:51:01.520239 6766 connection.cc:573] server connection from 127.0.71.65:40363 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:51:01.520202 6170 leader_election.cc:304] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32119a63667f414c9e4851a7a4a5db64, 4bea367d8bb04af49f22eaafc8bd4d03; no voters: 6c786aa789c24cdf89db361dfe2a28e1
I20260929 01:51:01.520633 6979 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 01:51:01.520768 6979 raft_consensus.cc:493] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 3 FOLLOWER]: Starting leader election (detected failure of leader 4bea367d8bb04af49f22eaafc8bd4d03)
I20260929 01:51:01.520889 6979 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 3 FOLLOWER]: Advancing to term 4
W20260929 01:51:01.520893 6170 connection.cc:573] server connection from 127.0.71.65:45897 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:51:01.521333 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:44677
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--webserver_port=43531
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:35355,127.0.71.85:34745,127.0.71.86:38995,127.0.71.87:44969,127.0.71.88:39357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {}
I20260929 01:51:01.523003 6979 raft_consensus.cc:515] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:51:01.523627 6979 leader_election.cc:290] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [CANDIDATE]: Term 4 election: Requested vote from peers 4bea367d8bb04af49f22eaafc8bd4d03 (127.0.71.126:43209), 6c786aa789c24cdf89db361dfe2a28e1 (127.0.71.124:38453)
I20260929 01:51:01.524009 6791 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32119a63667f414c9e4851a7a4a5db64" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4bea367d8bb04af49f22eaafc8bd4d03"
I20260929 01:51:01.524174 6280 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "32119a63667f414c9e4851a7a4a5db64" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "6c786aa789c24cdf89db361dfe2a28e1"
I20260929 01:51:01.524216 6791 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 3 FOLLOWER]: Advancing to term 4
I20260929 01:51:01.527971 6791 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 32119a63667f414c9e4851a7a4a5db64 in term 4.
I20260929 01:51:01.528400 6170 leader_election.cc:304] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 32119a63667f414c9e4851a7a4a5db64, 4bea367d8bb04af49f22eaafc8bd4d03; no voters: 6c786aa789c24cdf89db361dfe2a28e1
I20260929 01:51:01.528862 6979 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 4 FOLLOWER]: Leader election won for term 4
I20260929 01:51:01.529310 6979 raft_consensus.cc:697] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [term 4 LEADER]: Becoming Leader. State: Replica: 32119a63667f414c9e4851a7a4a5db64, State: Running, Role: LEADER
I20260929 01:51:01.529838 6979 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:51:01.531867 6983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32119a63667f414c9e4851a7a4a5db64. Latest consensus state: current_term: 4 leader_uuid: "32119a63667f414c9e4851a7a4a5db64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:51:01.532167 6983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [sys.catalog]: This master's current role is: LEADER
I20260929 01:51:01.532891 6985 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:51:01.534747 6985 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:51:01.535494 6985 catalog_manager.cc:1308] Loaded cluster ID: 8fc1f10b9eb142649bb4c0f1cb2126a6
I20260929 01:51:01.535611 6985 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:51:01.536731 6985 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:51:01.537482 6985 catalog_manager.cc:6230] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64: Loaded TSK: 0
I20260929 01:51:01.538308 6985 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:51:01.595049 6986 raft_consensus.cc:493] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 4bea367d8bb04af49f22eaafc8bd4d03)
I20260929 01:51:01.595316 6986 raft_consensus.cc:515] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } }
I20260929 01:51:01.597270 6986 leader_election.cc:290] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 4bea367d8bb04af49f22eaafc8bd4d03 (127.0.71.126:43209), 32119a63667f414c9e4851a7a4a5db64 (127.0.71.125:42769)
I20260929 01:51:01.597957 6196 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c786aa789c24cdf89db361dfe2a28e1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "32119a63667f414c9e4851a7a4a5db64" is_pre_election: true
I20260929 01:51:01.603668 6791 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "6c786aa789c24cdf89db361dfe2a28e1" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" is_pre_election: true
I20260929 01:51:01.604147 6791 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 6c786aa789c24cdf89db361dfe2a28e1 in current term 4: Already voted for candidate 32119a63667f414c9e4851a7a4a5db64 in this term.
I20260929 01:51:01.604970 6254 leader_election.cc:304] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 6c786aa789c24cdf89db361dfe2a28e1; no voters: 32119a63667f414c9e4851a7a4a5db64, 4bea367d8bb04af49f22eaafc8bd4d03
I20260929 01:51:01.605424 6986 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [term 3 FOLLOWER]: Advancing to term 4
I20260929 01:51:01.607503 6986 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 01:51:01.672081 6608 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43209
I20260929 01:51:01.672305 6608 heartbeater.cc:499] Master 127.0.71.126:43209 was elected leader, sending a full tablet report...
I20260929 01:51:01.673216 6781 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "417549d883ef467382ae5014881d55b4" instance_seqno: 1790646659626195) as {username='slave'} at 127.0.71.66:60767; Asking this server to re-register.
I20260929 01:51:01.673791 6608 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:01.674857 6781 ts_manager.cc:194] Registered new tserver with Master: 417549d883ef467382ae5014881d55b4 (127.0.71.66:34601)
W20260929 01:51:01.863946 6982 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:51:01.864341 6982 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:51:01.864531 6982 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:51:01.864631 6982 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:51:01.873135 6982 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:51:01.873385 6982 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:51:01.883499 6982 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35355,127.0.71.85:34745,127.0.71.86:38995,127.0.71.87:44969,127.0.71.88:39357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:44677
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=43531
--tserver_master_addrs=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:51:01.885243 6982 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:51:01.887109 6982 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:51:01.896652 6994 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:01.896620 6993 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:01.897192 6996 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:01.897583 6982 server_base.cc:1061] running on GCE node
I20260929 01:51:01.898129 6982 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:51:01.899222 6982 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:51:01.900535 6982 hybrid_clock.cc:648] HybridClock initialized: now 1790646661900335 us; error 104 us; skew 500 ppm
I20260929 01:51:01.900830 6982 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:51:01.902910 6982 webserver.cc:533] Webserver started at http://127.0.71.65:43531/ using document root <none> and password file <none>
I20260929 01:51:01.903605 6982 fs_manager.cc:362] Metadata directory not provided
I20260929 01:51:01.903805 6982 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:51:01.909282 6982 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:51:01.912070 7002 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:01.913612 6982 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:51:01.913801 6982 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "511afaa12e1c4e65a2c8347b29a2e5d1"
format_stamp: "Formatted at 2026-09-29 01:50:52 on dist-test-slave-vz23"
I20260929 01:51:01.914512 6982 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:51:01.951822 6982 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:51:01.952757 6982 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:51:01.953061 6982 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:51:01.954101 6982 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:51:01.955824 6982 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:51:01.955946 6982 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:01.956079 6982 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:51:01.956177 6982 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:51:01.956686 6982 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:51:01.956804 6982 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:51:01.956884 6982 fault_injection.cc:59] injecting fault for kudu (pid 6982): FLAGS_fault_before_start (process will exit)
I20260929 01:51:01.971976 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:44677
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--webserver_port=43531
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:35355,127.0.71.85:34745,127.0.71.86:38995,127.0.71.87:44969,127.0.71.88:39357
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:51:02.062846 6791 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [term 4 FOLLOWER]: Refusing update from remote peer 32119a63667f414c9e4851a7a4a5db64: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260929 01:51:02.063817 7011 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 01:51:02.067689 7012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32119a63667f414c9e4851a7a4a5db64. Latest consensus state: current_term: 4 leader_uuid: "32119a63667f414c9e4851a7a4a5db64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:51:02.067977 7012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:02.069422 7014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "32119a63667f414c9e4851a7a4a5db64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:51:02.069857 7014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [sys.catalog]: This master's current role is: LEADER
I20260929 01:51:02.070120 6280 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [term 4 FOLLOWER]: Refusing update from remote peer 32119a63667f414c9e4851a7a4a5db64: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260929 01:51:02.071086 7014 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 01:51:02.071595 7012 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "32119a63667f414c9e4851a7a4a5db64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:51:02.071777 7012 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4bea367d8bb04af49f22eaafc8bd4d03 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:02.080054 7020 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "32119a63667f414c9e4851a7a4a5db64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:51:02.080082 6986 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32119a63667f414c9e4851a7a4a5db64. Latest consensus state: current_term: 4 leader_uuid: "32119a63667f414c9e4851a7a4a5db64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:51:02.080585 7020 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:02.080740 6986 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6c786aa789c24cdf89db361dfe2a28e1 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:51:02.314792 7010 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:51:02.315109 7010 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:51:02.315294 7010 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:51:02.323698 7010 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:51:02.323930 7010 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:51:02.334149 7010 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35355,127.0.71.85:34745,127.0.71.86:38995,127.0.71.87:44969,127.0.71.88:39357
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:44677
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=43531
--tserver_master_addrs=127.0.71.126:43209,127.0.71.125:42769,127.0.71.124:38453
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:51:02.335757 7010 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:51:02.337517 7010 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:51:02.347123 7025 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:02.347177 7026 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:02.347452 7028 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:02.348318 7010 server_base.cc:1061] running on GCE node
I20260929 01:51:02.349000 7010 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:51:02.350104 7010 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:51:02.351310 7010 hybrid_clock.cc:648] HybridClock initialized: now 1790646662351185 us; error 91 us; skew 500 ppm
I20260929 01:51:02.351653 7010 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:51:02.353968 7010 webserver.cc:533] Webserver started at http://127.0.71.65:43531/ using document root <none> and password file <none>
I20260929 01:51:02.354715 7010 fs_manager.cc:362] Metadata directory not provided
I20260929 01:51:02.354826 7010 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:51:02.360553 7010 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:51:02.363533 7034 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:02.365015 7010 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:51:02.365182 7010 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "511afaa12e1c4e65a2c8347b29a2e5d1"
format_stamp: "Formatted at 2026-09-29 01:50:52 on dist-test-slave-vz23"
I20260929 01:51:02.365891 7010 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:51:02.385069 7010 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:51:02.385941 7010 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:51:02.386270 7010 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:51:02.387351 7010 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:51:02.389254 7010 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:51:02.389333 7010 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:02.389540 7010 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:51:02.389598 7010 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:02.428086 7010 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:44677
I20260929 01:51:02.428118 7146 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:44677 every 8 connection(s)
I20260929 01:51:02.429819 7010 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:51:02.431468 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 7010
I20260929 01:51:02.431865 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 7010
I20260929 01:51:02.452430 7142 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:51:02.537547 7014 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "32119a63667f414c9e4851a7a4a5db64" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4bea367d8bb04af49f22eaafc8bd4d03" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43209 } } peers { permanent_uuid: "32119a63667f414c9e4851a7a4a5db64" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42769 } } peers { permanent_uuid: "6c786aa789c24cdf89db361dfe2a28e1" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38453 } } }
I20260929 01:51:02.537819 7014 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32119a63667f414c9e4851a7a4a5db64 [sys.catalog]: This master's current role is: LEADER
I20260929 01:51:02.539635 7148 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43209
I20260929 01:51:02.540097 7148 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:02.541153 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 7010
I20260929 01:51:02.557601 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 6476
W20260929 01:51:02.558408 7158 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:51:02.442129 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:53795 (local address 127.0.71.124:38453)
0929 01:51:02.447763 (+ 5634us) server_negotiation.cc:207] Beginning negotiation
0929 01:51:02.447770 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0929 01:51:02.447794 (+ 24us) server_negotiation.cc:408] Connection header received
0929 01:51:02.447864 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:51:02.447871 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:51:02.447944 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:51:02.448062 (+ 118us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:51:02.449519 (+ 1457us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:51:02.452225 (+ 2706us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:51:02.557407 (+105182us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:53795: BlockingRecv error: recv got EOF from 127.0.71.65:53795 (error 108)
Metrics: {"server-negotiator.queue_time_us":5509,"thread_start_us":6923,"threads_started":1}
I20260929 01:51:02.566136 6602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:51:02.653795 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 6476
I20260929 01:51:02.671510 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 6614
I20260929 01:51:02.680197 6736 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:51:02.776470 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 6614
I20260929 01:51:02.793018 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 6749
I20260929 01:51:02.799731 6811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:51:02.906067 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 6749
I20260929 01:51:02.926332 6139 master_runner.cc:217] Error getting master consensus for 127.0.71.126:43209: Network error: recv got EOF from 127.0.71.126:43209 (error 108)
I20260929 01:51:02.926604 6139 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:51:02.927057 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 6139
I20260929 01:51:02.928151 6216 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:51:03.038560 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 6139
W20260929 01:51:03.058543 6256 connection.cc:573] server connection from 127.0.0.1:58682 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:51:03.059718 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 6223
I20260929 01:51:03.060933 6300 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:51:03.165948 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 6223
2026-09-29T01:51:03Z chronyd exiting
2026-09-29T01:51:03Z chronyd exiting
2026-09-29T01:51:03Z chronyd exiting
2026-09-29T01:51:03Z chronyd exiting
2026-09-29T01:51:03Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestBasicOperation/2 (13209 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/3
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/security.properties
2026-09-29 01:51:10: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260929 01:51:19.203894 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:39309
--webserver_interface=127.0.71.126
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723 with env {}
W20260929 01:51:19.534817 7443 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:51:19.535143 7443 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:51:19.535303 7443 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:51:19.543665 7443 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:51:19.543759 7443 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:51:19.543817 7443 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:51:19.543921 7443 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:51:19.554251 7443 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:39309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:51:19.555958 7443 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:51:19.557780 7443 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:51:19.566546 7449 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:19.566682 7448 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:19.567523 7451 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:19.567704 7443 server_base.cc:1061] running on GCE node
I20260929 01:51:19.568598 7443 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:51:19.568790 7443 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:51:19.568899 7443 hybrid_clock.cc:648] HybridClock initialized: now 1790646679568877 us; error 0 us; skew 500 ppm
I20260929 01:51:19.569296 7443 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:51:19.572026 7443 webserver.cc:533] Webserver started at http://127.0.71.126:41835/ using document root <none> and password file <none>
I20260929 01:51:19.572850 7443 fs_manager.cc:362] Metadata directory not provided
I20260929 01:51:19.572983 7443 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:51:19.573354 7443 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:51:19.575703 7443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "966a42cbe38040d6aed85c9aa2aec215"
format_stamp: "Formatted at 2026-09-29 01:51:19 on dist-test-slave-vz23"
I20260929 01:51:19.576565 7443 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "966a42cbe38040d6aed85c9aa2aec215"
format_stamp: "Formatted at 2026-09-29 01:51:19 on dist-test-slave-vz23"
I20260929 01:51:19.581765 7443 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 01:51:19.585207 7456 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:19.586612 7443 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:51:19.586781 7443 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "966a42cbe38040d6aed85c9aa2aec215"
format_stamp: "Formatted at 2026-09-29 01:51:19 on dist-test-slave-vz23"
I20260929 01:51:19.586982 7443 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:51:19.601991 7443 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:51:19.602921 7443 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:51:19.603207 7443 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:51:19.623746 7443 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:39309
I20260929 01:51:19.623788 7507 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:39309 every 8 connection(s)
I20260929 01:51:19.625291 7443 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:51:19.629618 7508 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:51:19.632896 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 7443
I20260929 01:51:19.633095 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:51:19.634338 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:37929
--webserver_interface=127.0.71.125
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723 with env {}
I20260929 01:51:19.637037 7508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:51:19.661947 7508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:51:19.664289 7458 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.125:37929: connect: Connection refused (error 111)
W20260929 01:51:19.666118 7508 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:37929: Network error: Client connection negotiation failed: client connection to 127.0.71.125:37929: connect: Connection refused (error 111)
I20260929 01:51:19.715476 7508 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } attempt: 1
W20260929 01:51:19.717865 7508 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:37929: Network error: Client connection negotiation failed: client connection to 127.0.71.125:37929: connect: Connection refused (error 111)
I20260929 01:51:19.786125 7508 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } attempt: 2
W20260929 01:51:19.788245 7508 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:37929: Network error: Client connection negotiation failed: client connection to 127.0.71.125:37929: connect: Connection refused (error 111)
I20260929 01:51:19.879508 7508 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } attempt: 3
W20260929 01:51:19.881660 7508 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:37929: Network error: Client connection negotiation failed: client connection to 127.0.71.125:37929: connect: Connection refused (error 111)
W20260929 01:51:19.970173 7510 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:51:19.970595 7510 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:51:19.970855 7510 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:51:19.980400 7510 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:51:19.980607 7510 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:51:19.980749 7510 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:51:19.980840 7510 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:51:19.992293 7510 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:37929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:51:19.994037 7510 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:51:19.995934 7510 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:51:20.004690 7519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:20.004658 7518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:20.005481 7521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:20.005959 7510 server_base.cc:1061] running on GCE node
I20260929 01:51:20.006767 7510 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:51:20.006970 7510 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:51:20.007066 7510 hybrid_clock.cc:648] HybridClock initialized: now 1790646680007043 us; error 0 us; skew 500 ppm
I20260929 01:51:20.007419 7510 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:51:20.010249 7510 webserver.cc:533] Webserver started at http://127.0.71.125:46207/ using document root <none> and password file <none>
I20260929 01:51:20.010979 7510 fs_manager.cc:362] Metadata directory not provided
I20260929 01:51:20.011111 7510 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:51:20.011462 7510 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:51:20.013871 7510 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/data/instance:
uuid: "291f81ab56bd492aa7755aa1bcf1564e"
format_stamp: "Formatted at 2026-09-29 01:51:20 on dist-test-slave-vz23"
I20260929 01:51:20.014616 7510 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/wal/instance:
uuid: "291f81ab56bd492aa7755aa1bcf1564e"
format_stamp: "Formatted at 2026-09-29 01:51:20 on dist-test-slave-vz23"
I20260929 01:51:20.019862 7510 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:51:20.023291 7526 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:20.024869 7510 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 01:51:20.024899 7508 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } attempt: 4
I20260929 01:51:20.025094 7510 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "291f81ab56bd492aa7755aa1bcf1564e"
format_stamp: "Formatted at 2026-09-29 01:51:20 on dist-test-slave-vz23"
I20260929 01:51:20.025337 7510 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
W20260929 01:51:20.027112 7508 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:37929: Network error: Client connection negotiation failed: client connection to 127.0.71.125:37929: connect: Connection refused (error 111)
I20260929 01:51:20.060814 7510 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:51:20.061888 7510 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:51:20.062228 7510 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:51:20.085281 7510 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:37929
I20260929 01:51:20.085299 7577 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:37929 every 8 connection(s)
I20260929 01:51:20.087124 7510 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:51:20.091538 7578 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:51:20.096432 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 7510
I20260929 01:51:20.096653 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:51:20.097927 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:43723
--webserver_interface=127.0.71.124
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723 with env {}
I20260929 01:51:20.098285 7578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:51:20.113332 7578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:51:20.123994 7578 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:51:20.125202 7528 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43723: connect: Connection refused (error 111)
W20260929 01:51:20.126972 7578 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43723: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43723: connect: Connection refused (error 111)
I20260929 01:51:20.176328 7578 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } attempt: 1
W20260929 01:51:20.178421 7578 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43723: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43723: connect: Connection refused (error 111)
I20260929 01:51:20.246672 7578 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } attempt: 2
W20260929 01:51:20.249138 7578 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43723: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43723: connect: Connection refused (error 111)
I20260929 01:51:20.326406 7508 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } attempt: 5
I20260929 01:51:20.333945 7508 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:51:20.336072 7508 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43723: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43723: connect: Connection refused (error 111)
I20260929 01:51:20.340391 7578 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } attempt: 3
W20260929 01:51:20.342310 7578 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43723: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43723: connect: Connection refused (error 111)
I20260929 01:51:20.387326 7508 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } attempt: 1
W20260929 01:51:20.389690 7508 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43723: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43723: connect: Connection refused (error 111)
W20260929 01:51:20.438547 7581 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:51:20.438983 7581 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:51:20.439122 7581 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:51:20.447768 7581 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:51:20.447916 7581 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:51:20.448019 7581 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:51:20.448102 7581 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:51:20.458014 7508 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } attempt: 2
I20260929 01:51:20.458987 7581 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:43723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
W20260929 01:51:20.460258 7508 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43723: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43723: connect: Connection refused (error 111)
I20260929 01:51:20.460877 7581 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:51:20.462632 7581 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:51:20.471556 7591 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:20.471487 7589 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:20.471583 7588 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:20.472856 7581 server_base.cc:1061] running on GCE node
I20260929 01:51:20.473637 7581 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:51:20.473830 7581 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:51:20.473945 7581 hybrid_clock.cc:648] HybridClock initialized: now 1790646680473922 us; error 0 us; skew 500 ppm
I20260929 01:51:20.474284 7581 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:51:20.476977 7581 webserver.cc:533] Webserver started at http://127.0.71.124:44789/ using document root <none> and password file <none>
I20260929 01:51:20.477746 7581 fs_manager.cc:362] Metadata directory not provided
I20260929 01:51:20.477866 7581 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:51:20.478240 7581 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:51:20.482338 7581 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/data/instance:
uuid: "d181618b1fb340f4aac54452bd2726d3"
format_stamp: "Formatted at 2026-09-29 01:51:20 on dist-test-slave-vz23"
I20260929 01:51:20.483121 7581 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/wal/instance:
uuid: "d181618b1fb340f4aac54452bd2726d3"
format_stamp: "Formatted at 2026-09-29 01:51:20 on dist-test-slave-vz23"
I20260929 01:51:20.485518 7578 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } attempt: 4
W20260929 01:51:20.487648 7578 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43723: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43723: connect: Connection refused (error 111)
I20260929 01:51:20.489145 7581 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 01:51:20.492810 7596 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:20.494210 7581 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 01:51:20.494382 7581 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "d181618b1fb340f4aac54452bd2726d3"
format_stamp: "Formatted at 2026-09-29 01:51:20 on dist-test-slave-vz23"
I20260929 01:51:20.494616 7581 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:51:20.520147 7581 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:51:20.521170 7581 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:51:20.521467 7581 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:51:20.542555 7581 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:43723
I20260929 01:51:20.542577 7647 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:43723 every 8 connection(s)
I20260929 01:51:20.544118 7581 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:51:20.548772 7648 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:51:20.549542 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 7581
I20260929 01:51:20.549747 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:51:20.556700 7648 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:51:20.566504 7508 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } attempt: 3
I20260929 01:51:20.570286 7648 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:51:20.577502 7648 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:51:20.580607 7508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: Bootstrap starting.
I20260929 01:51:20.584648 7508 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: Neither blocks nor log segments found. Creating new log.
I20260929 01:51:20.585968 7508 log.cc:826] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: Log is configured to *not* fsync() on all Append() calls
I20260929 01:51:20.588128 7648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3: Bootstrap starting.
I20260929 01:51:20.588910 7508 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: No bootstrap required, opened a new log
I20260929 01:51:20.591305 7648 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3: Neither blocks nor log segments found. Creating new log.
I20260929 01:51:20.592391 7648 log.cc:826] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3: Log is configured to *not* fsync() on all Append() calls
I20260929 01:51:20.594481 7508 raft_consensus.cc:359] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:20.594851 7508 raft_consensus.cc:385] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:51:20.595002 7508 raft_consensus.cc:740] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 966a42cbe38040d6aed85c9aa2aec215, State: Initialized, Role: FOLLOWER
I20260929 01:51:20.595165 7648 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3: No bootstrap required, opened a new log
I20260929 01:51:20.595691 7508 consensus_queue.cc:260] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:20.596724 7656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:20.597009 7656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:20.597723 7508 sys_catalog.cc:565] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:51:20.601583 7648 raft_consensus.cc:359] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:20.601923 7648 raft_consensus.cc:385] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:51:20.602049 7648 raft_consensus.cc:740] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d181618b1fb340f4aac54452bd2726d3, State: Initialized, Role: FOLLOWER
I20260929 01:51:20.602739 7648 consensus_queue.cc:260] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:20.603695 7663 sys_catalog.cc:455] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:20.603982 7663 sys_catalog.cc:458] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:20.605108 7648 sys_catalog.cc:565] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:51:20.605818 7462 tablet.cc:2433] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:51:20.611208 7668 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:51:20.612870 7668 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:51:20.613106 7443 master_runner.cc:187] Error getting master registration for 127.0.71.125:37929: OK, instance_id { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" instance_seqno: 1790646680078880 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:51:20.621528 7443 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:51:20.625734 7581 master_runner.cc:187] Error getting master registration for 127.0.71.125:37929: OK, instance_id { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" instance_seqno: 1790646680078880 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:51:20.635390 7581 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:51:20.636190 7680 catalog_manager.cc:1607] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:51:20.636328 7680 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:51:20.656708 7656 raft_consensus.cc:493] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:51:20.657018 7656 raft_consensus.cc:515] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:20.658926 7656 leader_election.cc:290] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 291f81ab56bd492aa7755aa1bcf1564e (127.0.71.125:37929), d181618b1fb340f4aac54452bd2726d3 (127.0.71.124:43723)
I20260929 01:51:20.660544 7623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "966a42cbe38040d6aed85c9aa2aec215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d181618b1fb340f4aac54452bd2726d3" is_pre_election: true
I20260929 01:51:20.661114 7623 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 966a42cbe38040d6aed85c9aa2aec215 in term 0.
I20260929 01:51:20.661912 7458 leader_election.cc:304] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 966a42cbe38040d6aed85c9aa2aec215, d181618b1fb340f4aac54452bd2726d3; no voters:
I20260929 01:51:20.662483 7656 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:51:20.662627 7656 raft_consensus.cc:493] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:51:20.662722 7656 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:51:20.666366 7656 raft_consensus.cc:515] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:20.667548 7656 leader_election.cc:290] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [CANDIDATE]: Term 1 election: Requested vote from peers 291f81ab56bd492aa7755aa1bcf1564e (127.0.71.125:37929), d181618b1fb340f4aac54452bd2726d3 (127.0.71.124:43723)
I20260929 01:51:20.668138 7623 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "966a42cbe38040d6aed85c9aa2aec215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d181618b1fb340f4aac54452bd2726d3"
I20260929 01:51:20.668345 7623 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:51:20.672048 7623 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 966a42cbe38040d6aed85c9aa2aec215 in term 1.
I20260929 01:51:20.672660 7458 leader_election.cc:304] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 966a42cbe38040d6aed85c9aa2aec215, d181618b1fb340f4aac54452bd2726d3; no voters:
I20260929 01:51:20.673071 7656 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:51:20.673717 7656 raft_consensus.cc:697] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 1 LEADER]: Becoming Leader. State: Replica: 966a42cbe38040d6aed85c9aa2aec215, State: Running, Role: LEADER
I20260929 01:51:20.674316 7656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:20.686355 7682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 966a42cbe38040d6aed85c9aa2aec215. Latest consensus state: current_term: 1 leader_uuid: "966a42cbe38040d6aed85c9aa2aec215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:20.686923 7682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: This master's current role is: LEADER
I20260929 01:51:20.688213 7684 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:51:20.690656 7684 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:51:20.700011 7623 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [term 1 FOLLOWER]: Refusing update from remote peer 966a42cbe38040d6aed85c9aa2aec215: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:51:20.701521 7682 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:51:20.726625 7663 sys_catalog.cc:455] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 966a42cbe38040d6aed85c9aa2aec215. Latest consensus state: current_term: 1 leader_uuid: "966a42cbe38040d6aed85c9aa2aec215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:20.726873 7663 sys_catalog.cc:458] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:20.729138 7685 mvcc.cc:204] Tried to move back new op lower bound from 7334488804134924288 to 7334488804047937536. Current Snapshot: MvccSnapshot[applied={T|T < 7334488804134924288}]
I20260929 01:51:20.730635 7663 sys_catalog.cc:455] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "966a42cbe38040d6aed85c9aa2aec215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:20.730855 7663 sys_catalog.cc:458] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:20.741216 7686 mvcc.cc:204] Tried to move back new op lower bound from 7334488804134924288 to 7334488804047937536. Current Snapshot: MvccSnapshot[applied={T|T < 7334488804134924288}]
I20260929 01:51:20.749571 7682 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "966a42cbe38040d6aed85c9aa2aec215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:20.749858 7682 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: This master's current role is: LEADER
I20260929 01:51:20.750823 7684 catalog_manager.cc:1396] Generated new cluster ID: 026ed1ecf44c40738ac2f6fc1984a6e8
I20260929 01:51:20.750931 7684 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:51:20.780861 7684 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 01:51:20.782651 7684 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:51:20.787321 7578 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } attempt: 5
I20260929 01:51:20.802728 7684 catalog_manager.cc:6219] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: Generated new TSK 0
I20260929 01:51:20.803875 7578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e: Bootstrap starting.
I20260929 01:51:20.804224 7684 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:51:20.814427 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--time_source=system_unsync with env {}
I20260929 01:51:20.818133 7578 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e: Neither blocks nor log segments found. Creating new log.
I20260929 01:51:20.821132 7578 log.cc:826] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e: Log is configured to *not* fsync() on all Append() calls
I20260929 01:51:20.835456 7578 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e: No bootstrap required, opened a new log
I20260929 01:51:20.855916 7578 raft_consensus.cc:359] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:20.856281 7578 raft_consensus.cc:385] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:51:20.856415 7578 raft_consensus.cc:740] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 291f81ab56bd492aa7755aa1bcf1564e, State: Initialized, Role: FOLLOWER
I20260929 01:51:20.857676 7578 consensus_queue.cc:260] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:20.872189 7692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:20.872713 7692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:20.873557 7578 sys_catalog.cc:565] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:51:20.873594 7553 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "966a42cbe38040d6aed85c9aa2aec215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "291f81ab56bd492aa7755aa1bcf1564e" is_pre_election: true
I20260929 01:51:20.874112 7553 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 966a42cbe38040d6aed85c9aa2aec215 in term 0.
I20260929 01:51:20.874866 7551 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:51:20.880028 7551 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [term 1 FOLLOWER]: Refusing update from remote peer 966a42cbe38040d6aed85c9aa2aec215: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:51:20.880535 7552 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "966a42cbe38040d6aed85c9aa2aec215" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "291f81ab56bd492aa7755aa1bcf1564e"
I20260929 01:51:20.885674 7656 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [LEADER]: Connected to new peer: Peer: permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:51:20.887631 7552 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 966a42cbe38040d6aed85c9aa2aec215 in term 1.
I20260929 01:51:20.927291 7692 sys_catalog.cc:455] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 966a42cbe38040d6aed85c9aa2aec215. Latest consensus state: current_term: 1 leader_uuid: "966a42cbe38040d6aed85c9aa2aec215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:20.927561 7692 sys_catalog.cc:458] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:20.927402 7699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "966a42cbe38040d6aed85c9aa2aec215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:20.927731 7699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:20.954972 7708 catalog_manager.cc:818] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260929 01:51:21.283933 7690 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:51:21.284256 7690 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:51:21.284402 7690 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:51:21.293025 7690 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:51:21.293227 7690 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:51:21.303324 7690 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:51:21.305078 7690 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:51:21.306825 7690 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:51:21.316025 7714 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:21.316345 7713 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:21.316896 7716 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:21.317368 7690 server_base.cc:1061] running on GCE node
I20260929 01:51:21.317888 7690 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:51:21.318025 7690 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:51:21.318085 7690 hybrid_clock.cc:648] HybridClock initialized: now 1790646681318069 us; error 0 us; skew 500 ppm
I20260929 01:51:21.318318 7690 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:51:21.320427 7690 webserver.cc:533] Webserver started at http://127.0.71.65:42003/ using document root <none> and password file <none>
I20260929 01:51:21.321138 7690 fs_manager.cc:362] Metadata directory not provided
I20260929 01:51:21.321221 7690 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:51:21.321506 7690 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:51:21.323704 7690 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "872fd31528444a2986ca624d0c8fa56e"
format_stamp: "Formatted at 2026-09-29 01:51:21 on dist-test-slave-vz23"
I20260929 01:51:21.324342 7690 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "872fd31528444a2986ca624d0c8fa56e"
format_stamp: "Formatted at 2026-09-29 01:51:21 on dist-test-slave-vz23"
I20260929 01:51:21.329291 7690 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 01:51:21.332396 7721 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:21.333729 7690 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:51:21.333880 7690 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "872fd31528444a2986ca624d0c8fa56e"
format_stamp: "Formatted at 2026-09-29 01:51:21 on dist-test-slave-vz23"
I20260929 01:51:21.334110 7690 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:51:21.349994 7690 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:51:21.350904 7690 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:51:21.351182 7690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:51:21.352128 7690 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:51:21.353762 7690 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:51:21.353848 7690 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:21.353995 7690 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:51:21.354099 7690 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:21.388952 7690 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:37677
I20260929 01:51:21.388985 7833 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:37677 every 8 connection(s)
I20260929 01:51:21.390383 7690 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:51:21.397171 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 7690
I20260929 01:51:21.397396 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
I20260929 01:51:21.415467 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.66:0
--local_ip_for_outbound_sockets=127.0.71.66
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--time_source=system_unsync with env {}
I20260929 01:51:21.428977 7835 heartbeater.cc:344] Connected to a master server at 127.0.71.126:39309
I20260929 01:51:21.429522 7835 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:21.430922 7835 heartbeater.cc:507] Master 127.0.71.126:39309 requested a full tablet report, sending...
I20260929 01:51:21.433907 7473 ts_manager.cc:194] Registered new tserver with Master: 872fd31528444a2986ca624d0c8fa56e (127.0.71.65:37677)
I20260929 01:51:21.436100 7473 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:58725
I20260929 01:51:21.440655 7836 heartbeater.cc:344] Connected to a master server at 127.0.71.125:37929
I20260929 01:51:21.440754 7834 heartbeater.cc:344] Connected to a master server at 127.0.71.124:43723
I20260929 01:51:21.440814 7836 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:21.440920 7834 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:21.441105 7836 heartbeater.cc:507] Master 127.0.71.125:37929 requested a full tablet report, sending...
I20260929 01:51:21.441280 7834 heartbeater.cc:507] Master 127.0.71.124:43723 requested a full tablet report, sending...
I20260929 01:51:21.442883 7543 ts_manager.cc:194] Registered new tserver with Master: 872fd31528444a2986ca624d0c8fa56e (127.0.71.65:37677)
I20260929 01:51:21.442950 7613 ts_manager.cc:194] Registered new tserver with Master: 872fd31528444a2986ca624d0c8fa56e (127.0.71.65:37677)
I20260929 01:51:21.493389 7656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "966a42cbe38040d6aed85c9aa2aec215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:21.493892 7656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: This master's current role is: LEADER
I20260929 01:51:21.638690 7680 catalog_manager.cc:1308] Loaded cluster ID: 026ed1ecf44c40738ac2f6fc1984a6e8
I20260929 01:51:21.638850 7680 catalog_manager.cc:1601] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3: loading cluster ID for follower catalog manager: success
I20260929 01:51:21.641568 7680 catalog_manager.cc:1623] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3: acquiring CA information for follower catalog manager: success
I20260929 01:51:21.643343 7680 catalog_manager.cc:1651] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 01:51:21.784015 7844 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:51:21.784358 7844 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:51:21.784574 7844 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:51:21.793581 7844 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:51:21.793749 7844 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.66
I20260929 01:51:21.803942 7844 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:51:21.805626 7844 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:51:21.807754 7844 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:51:21.818737 7853 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:21.818686 7852 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:21.819732 7855 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:21.819906 7844 server_base.cc:1061] running on GCE node
I20260929 01:51:21.820593 7844 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:51:21.820734 7844 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:51:21.820793 7844 hybrid_clock.cc:648] HybridClock initialized: now 1790646681820776 us; error 0 us; skew 500 ppm
I20260929 01:51:21.821034 7844 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:51:21.823263 7844 webserver.cc:533] Webserver started at http://127.0.71.66:33181/ using document root <none> and password file <none>
I20260929 01:51:21.823988 7844 fs_manager.cc:362] Metadata directory not provided
I20260929 01:51:21.824074 7844 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:51:21.824357 7844 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:51:21.826841 7844 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/data/instance:
uuid: "f4251826e5f34caf8ff26cfb99beb84e"
format_stamp: "Formatted at 2026-09-29 01:51:21 on dist-test-slave-vz23"
I20260929 01:51:21.827523 7844 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/wal/instance:
uuid: "f4251826e5f34caf8ff26cfb99beb84e"
format_stamp: "Formatted at 2026-09-29 01:51:21 on dist-test-slave-vz23"
I20260929 01:51:21.833292 7844 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 01:51:21.836508 7860 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:21.837951 7844 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:51:21.838120 7844 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/wal
uuid: "f4251826e5f34caf8ff26cfb99beb84e"
format_stamp: "Formatted at 2026-09-29 01:51:21 on dist-test-slave-vz23"
I20260929 01:51:21.838336 7844 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:51:21.850389 7844 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:51:21.851164 7844 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:51:21.851387 7844 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:51:21.852218 7844 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:51:21.854401 7844 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:51:21.854485 7844 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:21.854609 7844 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:51:21.854698 7844 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:21.891171 7844 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.66:34735
I20260929 01:51:21.891206 7972 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.66:34735 every 8 connection(s)
I20260929 01:51:21.892755 7844 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
I20260929 01:51:21.893428 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 7844
I20260929 01:51:21.893699 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/wal/instance
I20260929 01:51:21.899413 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.67:0
--local_ip_for_outbound_sockets=127.0.71.67
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--time_source=system_unsync with env {}
I20260929 01:51:21.913996 7973 heartbeater.cc:344] Connected to a master server at 127.0.71.124:43723
I20260929 01:51:21.914420 7973 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:21.915442 7973 heartbeater.cc:507] Master 127.0.71.124:43723 requested a full tablet report, sending...
I20260929 01:51:21.916178 7975 heartbeater.cc:344] Connected to a master server at 127.0.71.125:37929
I20260929 01:51:21.916337 7975 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:21.916929 7975 heartbeater.cc:507] Master 127.0.71.125:37929 requested a full tablet report, sending...
I20260929 01:51:21.917506 7974 heartbeater.cc:344] Connected to a master server at 127.0.71.126:39309
I20260929 01:51:21.917642 7974 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:21.917725 7613 ts_manager.cc:194] Registered new tserver with Master: f4251826e5f34caf8ff26cfb99beb84e (127.0.71.66:34735)
I20260929 01:51:21.918110 7974 heartbeater.cc:507] Master 127.0.71.126:39309 requested a full tablet report, sending...
I20260929 01:51:21.918558 7543 ts_manager.cc:194] Registered new tserver with Master: f4251826e5f34caf8ff26cfb99beb84e (127.0.71.66:34735)
I20260929 01:51:21.919010 7473 ts_manager.cc:194] Registered new tserver with Master: f4251826e5f34caf8ff26cfb99beb84e (127.0.71.66:34735)
I20260929 01:51:21.919852 7473 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.66:35267
I20260929 01:51:21.959935 7708 catalog_manager.cc:1308] Loaded cluster ID: 026ed1ecf44c40738ac2f6fc1984a6e8
I20260929 01:51:21.960078 7708 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e: loading cluster ID for follower catalog manager: success
I20260929 01:51:21.963997 7708 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e: acquiring CA information for follower catalog manager: success
I20260929 01:51:21.966459 7708 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 01:51:22.252324 7980 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:51:22.252744 7980 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:51:22.252898 7980 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:51:22.261612 7980 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:51:22.261845 7980 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.67
I20260929 01:51:22.272205 7980 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:51:22.274049 7980 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:51:22.275847 7980 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:51:22.285678 7986 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:22.286064 7987 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:22.287143 7989 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:22.288241 7980 server_base.cc:1061] running on GCE node
I20260929 01:51:22.288903 7980 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:51:22.289078 7980 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:51:22.289173 7980 hybrid_clock.cc:648] HybridClock initialized: now 1790646682289154 us; error 0 us; skew 500 ppm
I20260929 01:51:22.289561 7980 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:51:22.291831 7980 webserver.cc:533] Webserver started at http://127.0.71.67:35169/ using document root <none> and password file <none>
I20260929 01:51:22.292692 7980 fs_manager.cc:362] Metadata directory not provided
I20260929 01:51:22.292783 7980 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:51:22.293133 7980 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:51:22.296059 7980 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/data/instance:
uuid: "d1e530562ffb4eff970aca2d4bed29c1"
format_stamp: "Formatted at 2026-09-29 01:51:22 on dist-test-slave-vz23"
I20260929 01:51:22.296839 7980 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/wal/instance:
uuid: "d1e530562ffb4eff970aca2d4bed29c1"
format_stamp: "Formatted at 2026-09-29 01:51:22 on dist-test-slave-vz23"
I20260929 01:51:22.302631 7980 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260929 01:51:22.306630 7994 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:22.308089 7980 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.001s
I20260929 01:51:22.308241 7980 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/wal
uuid: "d1e530562ffb4eff970aca2d4bed29c1"
format_stamp: "Formatted at 2026-09-29 01:51:22 on dist-test-slave-vz23"
I20260929 01:51:22.308501 7980 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:51:22.346791 7980 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:51:22.347736 7980 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:51:22.348016 7980 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:51:22.349586 7980 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:51:22.351357 7980 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:51:22.351482 7980 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:22.351609 7980 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:51:22.351686 7980 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:22.388790 7980 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.67:33719
I20260929 01:51:22.388875 8106 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.67:33719 every 8 connection(s)
I20260929 01:51:22.390210 7980 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
I20260929 01:51:22.396410 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 7980
I20260929 01:51:22.396654 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/wal/instance
I20260929 01:51:22.410014 8108 heartbeater.cc:344] Connected to a master server at 127.0.71.126:39309
I20260929 01:51:22.410538 8108 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:22.411821 8108 heartbeater.cc:507] Master 127.0.71.126:39309 requested a full tablet report, sending...
I20260929 01:51:22.414242 7473 ts_manager.cc:194] Registered new tserver with Master: d1e530562ffb4eff970aca2d4bed29c1 (127.0.71.67:33719)
I20260929 01:51:22.415266 7473 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.67:36817
I20260929 01:51:22.416163 8107 heartbeater.cc:344] Connected to a master server at 127.0.71.124:43723
I20260929 01:51:22.416216 8109 heartbeater.cc:344] Connected to a master server at 127.0.71.125:37929
I20260929 01:51:22.416307 8107 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:22.416396 8109 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:22.416968 8107 heartbeater.cc:507] Master 127.0.71.124:43723 requested a full tablet report, sending...
I20260929 01:51:22.417047 8109 heartbeater.cc:507] Master 127.0.71.125:37929 requested a full tablet report, sending...
I20260929 01:51:22.419485 7613 ts_manager.cc:194] Registered new tserver with Master: d1e530562ffb4eff970aca2d4bed29c1 (127.0.71.67:33719)
I20260929 01:51:22.419488 7543 ts_manager.cc:194] Registered new tserver with Master: d1e530562ffb4eff970aca2d4bed29c1 (127.0.71.67:33719)
I20260929 01:51:22.427518 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:51:22.435196 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:51:22.435403 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 7443
I20260929 01:51:22.436321 7503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:51:22.440555 7835 heartbeater.cc:499] Master 127.0.71.126:39309 was elected leader, sending a full tablet report...
I20260929 01:51:22.552829 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 7443
W20260929 01:51:22.573073 7527 connection.cc:573] server connection from 127.0.0.1:41886 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:51:22.573887 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:39309
--webserver_interface=127.0.71.126
--webserver_port=41835
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723 with env {}
W20260929 01:51:22.905017 8115 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:51:22.905385 8115 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:51:22.905517 8115 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:51:22.914924 8115 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:51:22.915050 8115 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:51:22.915143 8115 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:51:22.915257 8115 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:51:22.921913 7974 heartbeater.cc:499] Master 127.0.71.126:39309 was elected leader, sending a full tablet report...
W20260929 01:51:22.923496 7974 heartbeater.cc:646] Failed to heartbeat to 127.0.71.126:39309 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.71.126:39309: connect: Connection refused (error 111)
I20260929 01:51:22.927403 8115 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:39309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=41835
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:51:22.928992 8115 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:51:22.930712 8115 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:51:22.939175 8121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:22.939278 8124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:22.939232 8122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:22.941431 8115 server_base.cc:1061] running on GCE node
I20260929 01:51:22.942260 8115 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:51:22.942456 8115 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:51:22.942539 8115 hybrid_clock.cc:648] HybridClock initialized: now 1790646682942520 us; error 0 us; skew 500 ppm
I20260929 01:51:22.942883 8115 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:51:22.946004 8115 webserver.cc:533] Webserver started at http://127.0.71.126:41835/ using document root <none> and password file <none>
I20260929 01:51:22.946816 8115 fs_manager.cc:362] Metadata directory not provided
I20260929 01:51:22.946972 8115 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:51:22.953037 8115 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:51:22.956230 8129 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:22.957758 8115 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:51:22.957909 8115 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "966a42cbe38040d6aed85c9aa2aec215"
format_stamp: "Formatted at 2026-09-29 01:51:19 on dist-test-slave-vz23"
I20260929 01:51:22.958621 8115 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:51:22.973340 8115 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:51:22.974224 8115 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:51:22.974475 8115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:51:22.995649 8115 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:39309
I20260929 01:51:22.995800 8180 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:39309 every 8 connection(s)
I20260929 01:51:22.997378 8115 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:51:23.002770 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 8115
I20260929 01:51:23.003194 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 7690
I20260929 01:51:23.005674 8181 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:51:23.011799 8181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: Bootstrap starting.
I20260929 01:51:23.017107 7829 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:51:23.035655 8181 log.cc:826] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: Log is configured to *not* fsync() on all Append() calls
I20260929 01:51:23.046368 8181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:51:23.047024 8181 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: Bootstrap complete.
I20260929 01:51:23.057160 8181 raft_consensus.cc:359] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:23.058140 8181 raft_consensus.cc:740] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 966a42cbe38040d6aed85c9aa2aec215, State: Initialized, Role: FOLLOWER
I20260929 01:51:23.059000 8181 consensus_queue.cc:260] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:23.059863 8187 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:23.060143 8187 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:23.061199 8181 sys_catalog.cc:565] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:51:23.078931 8198 catalog_manager.cc:1308] Loaded cluster ID: 026ed1ecf44c40738ac2f6fc1984a6e8
I20260929 01:51:23.080910 8198 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: loading cluster ID for follower catalog manager: success
I20260929 01:51:23.094905 8198 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: acquiring CA information for follower catalog manager: success
I20260929 01:51:23.106698 8198 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:51:23.127969 8115 master_runner.cc:264] No leader master found from master 966a42cbe38040d6aed85c9aa2aec215
I20260929 01:51:23.128137 8115 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:51:23.143469 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 7690
I20260929 01:51:23.160161 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:37677
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--webserver_port=42003
--webserver_interface=127.0.71.65
--time_source=system_unsync with env {}
I20260929 01:51:23.418946 8108 heartbeater.cc:499] Master 127.0.71.126:39309 was elected leader, sending a full tablet report...
I20260929 01:51:23.431130 8146 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d1e530562ffb4eff970aca2d4bed29c1" instance_seqno: 1790646682379977) as {username='slave'} at 127.0.71.67:55781; Asking this server to re-register.
I20260929 01:51:23.432108 8108 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:23.433977 8146 ts_manager.cc:194] Registered new tserver with Master: d1e530562ffb4eff970aca2d4bed29c1 (127.0.71.67:33719)
W20260929 01:51:23.500600 8204 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:51:23.500959 8204 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:51:23.501096 8204 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:51:23.509979 8204 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:51:23.510216 8204 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:51:23.520486 8204 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:37677
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=42003
--tserver_master_addrs=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:51:23.522151 8204 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:51:23.523981 8204 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:51:23.534580 8210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:23.534631 8211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:23.535135 8213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:23.535835 8204 server_base.cc:1061] running on GCE node
I20260929 01:51:23.536494 8204 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:51:23.536674 8204 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:51:23.536778 8204 hybrid_clock.cc:648] HybridClock initialized: now 1790646683536742 us; error 0 us; skew 500 ppm
I20260929 01:51:23.537065 8204 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:51:23.539288 8204 webserver.cc:533] Webserver started at http://127.0.71.65:42003/ using document root <none> and password file <none>
I20260929 01:51:23.540028 8204 fs_manager.cc:362] Metadata directory not provided
I20260929 01:51:23.540151 8204 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:51:23.546473 8204 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 01:51:23.549958 8218 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:23.551651 8204 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:51:23.551822 8204 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "872fd31528444a2986ca624d0c8fa56e"
format_stamp: "Formatted at 2026-09-29 01:51:21 on dist-test-slave-vz23"
I20260929 01:51:23.552732 8204 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:51:23.580077 8204 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:51:23.581056 8204 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:51:23.581346 8204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:51:23.582266 8204 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:51:23.583897 8204 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:51:23.583999 8204 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:23.584139 8204 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:51:23.584208 8204 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:23.620705 8204 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:37677
I20260929 01:51:23.620755 8330 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:37677 every 8 connection(s)
I20260929 01:51:23.622217 8204 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:51:23.631527 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 8204
I20260929 01:51:23.639552 8332 heartbeater.cc:344] Connected to a master server at 127.0.71.126:39309
I20260929 01:51:23.639839 8331 heartbeater.cc:344] Connected to a master server at 127.0.71.124:43723
I20260929 01:51:23.639966 8332 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:23.639986 8331 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:23.640153 8333 heartbeater.cc:344] Connected to a master server at 127.0.71.125:37929
I20260929 01:51:23.641459 8331 heartbeater.cc:507] Master 127.0.71.124:43723 requested a full tablet report, sending...
I20260929 01:51:23.641433 8332 heartbeater.cc:507] Master 127.0.71.126:39309 requested a full tablet report, sending...
I20260929 01:51:23.642761 8333 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:23.643254 8333 heartbeater.cc:507] Master 127.0.71.125:37929 requested a full tablet report, sending...
I20260929 01:51:23.643617 7613 ts_manager.cc:194] Re-registered known tserver with Master: 872fd31528444a2986ca624d0c8fa56e (127.0.71.65:37677)
I20260929 01:51:23.644551 7543 ts_manager.cc:194] Re-registered known tserver with Master: 872fd31528444a2986ca624d0c8fa56e (127.0.71.65:37677)
I20260929 01:51:23.644729 8146 ts_manager.cc:194] Registered new tserver with Master: 872fd31528444a2986ca624d0c8fa56e (127.0.71.65:37677)
I20260929 01:51:23.667135 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 8204
I20260929 01:51:23.689042 8326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:51:23.776520 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 8204
I20260929 01:51:23.793460 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:37677
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--webserver_port=42003
--webserver_interface=127.0.71.65
--time_source=system_unsync
--fault_before_start=1.0 with env {}
I20260929 01:51:23.796386 8345 raft_consensus.cc:493] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 966a42cbe38040d6aed85c9aa2aec215)
I20260929 01:51:23.796772 8345 raft_consensus.cc:515] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:23.798205 8345 leader_election.cc:290] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 966a42cbe38040d6aed85c9aa2aec215 (127.0.71.126:39309), 291f81ab56bd492aa7755aa1bcf1564e (127.0.71.125:37929)
I20260929 01:51:23.798880 7551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d181618b1fb340f4aac54452bd2726d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "291f81ab56bd492aa7755aa1bcf1564e" is_pre_election: true
I20260929 01:51:23.806464 8156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d181618b1fb340f4aac54452bd2726d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "966a42cbe38040d6aed85c9aa2aec215" is_pre_election: true
I20260929 01:51:23.807035 8156 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d181618b1fb340f4aac54452bd2726d3 in term 1.
I20260929 01:51:23.807715 7597 leader_election.cc:304] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 966a42cbe38040d6aed85c9aa2aec215, d181618b1fb340f4aac54452bd2726d3; no voters: 291f81ab56bd492aa7755aa1bcf1564e
I20260929 01:51:23.808122 8345 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 01:51:23.808252 8345 raft_consensus.cc:493] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 966a42cbe38040d6aed85c9aa2aec215)
I20260929 01:51:23.808369 8345 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:51:23.810495 8345 raft_consensus.cc:515] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:23.811215 8345 leader_election.cc:290] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [CANDIDATE]: Term 2 election: Requested vote from peers 966a42cbe38040d6aed85c9aa2aec215 (127.0.71.126:39309), 291f81ab56bd492aa7755aa1bcf1564e (127.0.71.125:37929)
I20260929 01:51:23.811609 8156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d181618b1fb340f4aac54452bd2726d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "966a42cbe38040d6aed85c9aa2aec215"
I20260929 01:51:23.811837 8156 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:51:23.812079 7551 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d181618b1fb340f4aac54452bd2726d3" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "291f81ab56bd492aa7755aa1bcf1564e"
I20260929 01:51:23.815841 8156 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d181618b1fb340f4aac54452bd2726d3 in term 2.
I20260929 01:51:23.816298 7597 leader_election.cc:304] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 966a42cbe38040d6aed85c9aa2aec215, d181618b1fb340f4aac54452bd2726d3; no voters: 291f81ab56bd492aa7755aa1bcf1564e
I20260929 01:51:23.816633 8345 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:51:23.817044 8345 raft_consensus.cc:697] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [term 2 LEADER]: Becoming Leader. State: Replica: d181618b1fb340f4aac54452bd2726d3, State: Running, Role: LEADER
I20260929 01:51:23.817484 8345 consensus_queue.cc:237] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:23.819244 8348 sys_catalog.cc:455] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d181618b1fb340f4aac54452bd2726d3. Latest consensus state: current_term: 2 leader_uuid: "d181618b1fb340f4aac54452bd2726d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:23.819561 8348 sys_catalog.cc:458] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [sys.catalog]: This master's current role is: LEADER
I20260929 01:51:23.820324 8350 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:51:23.822131 8350 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:51:23.822923 8350 catalog_manager.cc:1308] Loaded cluster ID: 026ed1ecf44c40738ac2f6fc1984a6e8
I20260929 01:51:23.823024 8350 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:51:23.824119 8350 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:51:23.824976 8350 catalog_manager.cc:6230] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3: Loaded TSK: 0
I20260929 01:51:23.825769 8350 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:51:23.931694 7974 heartbeater.cc:344] Connected to a master server at 127.0.71.126:39309
I20260929 01:51:23.931928 7974 heartbeater.cc:499] Master 127.0.71.126:39309 was elected leader, sending a full tablet report...
I20260929 01:51:23.933094 8146 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f4251826e5f34caf8ff26cfb99beb84e" instance_seqno: 1790646681882195) as {username='slave'} at 127.0.71.66:40211; Asking this server to re-register.
I20260929 01:51:23.933844 7974 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:23.935082 8146 ts_manager.cc:194] Registered new tserver with Master: f4251826e5f34caf8ff26cfb99beb84e (127.0.71.66:34735)
I20260929 01:51:24.053911 8352 raft_consensus.cc:493] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 966a42cbe38040d6aed85c9aa2aec215)
I20260929 01:51:24.054181 8352 raft_consensus.cc:515] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:24.055701 8352 leader_election.cc:290] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 966a42cbe38040d6aed85c9aa2aec215 (127.0.71.126:39309), d181618b1fb340f4aac54452bd2726d3 (127.0.71.124:43723)
I20260929 01:51:24.056833 7622 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "291f81ab56bd492aa7755aa1bcf1564e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d181618b1fb340f4aac54452bd2726d3" is_pre_election: true
I20260929 01:51:24.062171 8156 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "291f81ab56bd492aa7755aa1bcf1564e" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "966a42cbe38040d6aed85c9aa2aec215" is_pre_election: true
I20260929 01:51:24.062557 8156 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 291f81ab56bd492aa7755aa1bcf1564e in current term 2: Already voted for candidate d181618b1fb340f4aac54452bd2726d3 in this term.
I20260929 01:51:24.063295 7527 leader_election.cc:304] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 291f81ab56bd492aa7755aa1bcf1564e; no voters: 966a42cbe38040d6aed85c9aa2aec215, d181618b1fb340f4aac54452bd2726d3
I20260929 01:51:24.063818 8352 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:51:24.066157 8352 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260929 01:51:24.151321 8346 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:51:24.151969 8346 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:51:24.152077 8346 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:51:24.152136 8346 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:51:24.166579 8346 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:51:24.166878 8346 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:51:24.185890 8346 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:37677
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=42003
--tserver_master_addrs=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:51:24.188303 8346 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:51:24.190146 8346 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:51:24.200793 8359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:24.201406 8358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:24.201984 8361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:24.202593 8346 server_base.cc:1061] running on GCE node
I20260929 01:51:24.203269 8346 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:51:24.203430 8346 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:51:24.203558 8346 hybrid_clock.cc:648] HybridClock initialized: now 1790646684203499 us; error 0 us; skew 500 ppm
I20260929 01:51:24.203917 8346 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:51:24.206444 8346 webserver.cc:533] Webserver started at http://127.0.71.65:42003/ using document root <none> and password file <none>
I20260929 01:51:24.207238 8346 fs_manager.cc:362] Metadata directory not provided
I20260929 01:51:24.207372 8346 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:51:24.213122 8346 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:51:24.215921 8366 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:24.217305 8346 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:51:24.217492 8346 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "872fd31528444a2986ca624d0c8fa56e"
format_stamp: "Formatted at 2026-09-29 01:51:21 on dist-test-slave-vz23"
I20260929 01:51:24.218209 8346 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:51:24.243165 8156 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 2 FOLLOWER]: Refusing update from remote peer d181618b1fb340f4aac54452bd2726d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 01:51:24.244107 8348 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:51:24.247888 8371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d181618b1fb340f4aac54452bd2726d3. Latest consensus state: current_term: 2 leader_uuid: "d181618b1fb340f4aac54452bd2726d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:24.248165 8371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:24.249186 8348 sys_catalog.cc:455] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "d181618b1fb340f4aac54452bd2726d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:24.249509 8348 sys_catalog.cc:458] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [sys.catalog]: This master's current role is: LEADER
I20260929 01:51:24.250659 7551 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [term 2 FOLLOWER]: Refusing update from remote peer d181618b1fb340f4aac54452bd2726d3: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 01:51:24.251590 8348 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [LEADER]: Connected to new peer: Peer: permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:51:24.262228 8371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "d181618b1fb340f4aac54452bd2726d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:24.262462 8371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:24.264107 8378 sys_catalog.cc:455] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "d181618b1fb340f4aac54452bd2726d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:24.264166 8352 sys_catalog.cc:455] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [sys.catalog]: SysCatalogTable state changed. Reason: New leader d181618b1fb340f4aac54452bd2726d3. Latest consensus state: current_term: 2 leader_uuid: "d181618b1fb340f4aac54452bd2726d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:24.264430 8352 sys_catalog.cc:458] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:24.264600 8378 sys_catalog.cc:458] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:24.270702 8346 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:51:24.271724 8346 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:51:24.272001 8346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:51:24.273165 8346 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:51:24.274875 8346 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:51:24.275005 8346 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:24.275134 8346 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:51:24.275243 8346 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:51:24.275648 8346 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:51:24.275784 8346 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:51:24.275858 8346 fault_injection.cc:59] injecting fault for kudu (pid 8346): FLAGS_fault_before_start (process will exit)
I20260929 01:51:24.296378 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:37677
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--webserver_port=42003
--webserver_interface=127.0.71.65
--time_source=system_unsync with env {}
W20260929 01:51:24.635335 8382 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:51:24.635699 8382 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:51:24.635847 8382 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:51:24.644344 8382 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:51:24.644600 8382 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:51:24.655031 8382 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:37677
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=42003
--tserver_master_addrs=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:51:24.656697 8382 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:51:24.658541 8382 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:51:24.670575 8387 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:24.671970 8374 sys_catalog.cc:455] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "d181618b1fb340f4aac54452bd2726d3" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:24.672228 8374 sys_catalog.cc:458] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [sys.catalog]: This master's current role is: LEADER
W20260929 01:51:24.672752 8388 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:24.675223 8390 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:24.676841 8382 server_base.cc:1061] running on GCE node
I20260929 01:51:24.677474 8382 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:51:24.677663 8382 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:51:24.677757 8382 hybrid_clock.cc:648] HybridClock initialized: now 1790646684677739 us; error 0 us; skew 500 ppm
I20260929 01:51:24.678076 8382 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:51:24.680430 8382 webserver.cc:533] Webserver started at http://127.0.71.65:42003/ using document root <none> and password file <none>
I20260929 01:51:24.681324 8382 fs_manager.cc:362] Metadata directory not provided
I20260929 01:51:24.681418 8382 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:51:24.687285 8382 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 01:51:24.690203 8395 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:24.691617 8382 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:51:24.691895 8382 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "872fd31528444a2986ca624d0c8fa56e"
format_stamp: "Formatted at 2026-09-29 01:51:21 on dist-test-slave-vz23"
I20260929 01:51:24.692684 8382 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:51:24.712280 8382 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:51:24.713370 8382 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:51:24.713663 8382 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:51:24.714731 8382 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:51:24.716591 8382 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:51:24.716701 8382 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:24.716845 8382 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:51:24.716915 8382 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:24.754235 8382 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:37677
I20260929 01:51:24.754328 8507 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:37677 every 8 connection(s)
I20260929 01:51:24.755832 8382 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:51:24.756814 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 8382
I20260929 01:51:24.757202 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 8382
I20260929 01:51:24.793309 8509 heartbeater.cc:344] Connected to a master server at 127.0.71.126:39309
I20260929 01:51:24.793756 8509 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:24.794239 8503 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:51:24.883558 8508 heartbeater.cc:344] Connected to a master server at 127.0.71.124:43723
I20260929 01:51:24.883564 8510 heartbeater.cc:344] Connected to a master server at 127.0.71.125:37929
I20260929 01:51:24.883940 8508 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:24.884291 8510 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:24.884711 8509 heartbeater.cc:507] Master 127.0.71.126:39309 requested a full tablet report, sending...
I20260929 01:51:24.884893 8510 heartbeater.cc:507] Master 127.0.71.125:37929 requested a full tablet report, sending...
I20260929 01:51:24.885303 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 8382
I20260929 01:51:24.885334 8508 heartbeater.cc:507] Master 127.0.71.124:43723 requested a full tablet report, sending...
I20260929 01:51:24.901546 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 7844
I20260929 01:51:24.910368 7968 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:51:24.998788 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 7844
I20260929 01:51:24.998780 7973 heartbeater.cc:499] Master 127.0.71.124:43723 was elected leader, sending a full tablet report...
I20260929 01:51:25.019021 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 7980
I20260929 01:51:25.027206 8102 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:51:25.121193 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 7980
I20260929 01:51:25.137434 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 8115
I20260929 01:51:25.144076 8176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:51:25.250031 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 8115
I20260929 01:51:25.269666 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 7510
I20260929 01:51:25.270859 7573 generic_service.cc:196] Checking for leaks (request via RPC)
W20260929 01:51:25.341423 7597 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.126:39309: connect: Connection refused (error 111)
W20260929 01:51:25.349663 7597 consensus_peers.cc:597] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 -> Peer 966a42cbe38040d6aed85c9aa2aec215 (127.0.71.126:39309): Couldn't send request to peer 966a42cbe38040d6aed85c9aa2aec215. Status: Network error: Client connection negotiation failed: client connection to 127.0.71.126:39309: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 01:51:25.378387 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 7510
I20260929 01:51:25.397620 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 7581
I20260929 01:51:25.398826 7643 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:51:25.509138 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 7581
Shutting down hive metastore.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/security.properties
2026-09-29 01:51:26: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260929 01:51:34.935436 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:39309
--webserver_interface=127.0.71.126
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723 with env {}
W20260929 01:51:35.264187 8710 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:51:35.264530 8710 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:51:35.264598 8710 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:51:35.272845 8710 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:51:35.272931 8710 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:51:35.272986 8710 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:51:35.273029 8710 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:51:35.283355 8710 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:39309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:51:35.284982 8710 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:51:35.286717 8710 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:51:35.295269 8716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:35.295276 8715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:35.295567 8710 server_base.cc:1061] running on GCE node
W20260929 01:51:35.295912 8718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:35.296864 8710 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:51:35.297053 8710 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:51:35.297161 8710 hybrid_clock.cc:648] HybridClock initialized: now 1790646695297123 us; error 0 us; skew 500 ppm
I20260929 01:51:35.297487 8710 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:51:35.300079 8710 webserver.cc:533] Webserver started at http://127.0.71.126:36047/ using document root <none> and password file <none>
I20260929 01:51:35.300943 8710 fs_manager.cc:362] Metadata directory not provided
I20260929 01:51:35.301064 8710 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:51:35.306361 8710 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 01:51:35.308995 8723 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:35.310348 8710 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:51:35.310530 8710 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "966a42cbe38040d6aed85c9aa2aec215"
format_stamp: "Formatted at 2026-09-29 01:51:19 on dist-test-slave-vz23"
I20260929 01:51:35.311245 8710 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:51:35.323278 8710 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:51:35.324137 8710 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:51:35.324432 8710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:51:35.345006 8710 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:39309
I20260929 01:51:35.345006 8774 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:39309 every 8 connection(s)
I20260929 01:51:35.346765 8710 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:51:35.353910 8775 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:51:35.353956 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 8710
I20260929 01:51:35.354108 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:51:35.355268 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:37929
--webserver_interface=127.0.71.125
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723 with env {}
I20260929 01:51:35.358881 8775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: Bootstrap starting.
I20260929 01:51:35.376013 8775 log.cc:826] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: Log is configured to *not* fsync() on all Append() calls
I20260929 01:51:35.385296 8775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:51:35.385902 8775 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: Bootstrap complete.
I20260929 01:51:35.392678 8775 raft_consensus.cc:359] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:35.393347 8775 raft_consensus.cc:740] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 966a42cbe38040d6aed85c9aa2aec215, State: Initialized, Role: FOLLOWER
I20260929 01:51:35.394100 8775 consensus_queue.cc:260] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:35.395058 8779 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:35.395340 8779 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:35.396072 8775 sys_catalog.cc:565] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:51:35.406931 8790 catalog_manager.cc:1308] Loaded cluster ID: 026ed1ecf44c40738ac2f6fc1984a6e8
I20260929 01:51:35.407157 8790 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: loading cluster ID for follower catalog manager: success
I20260929 01:51:35.414168 8790 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: acquiring CA information for follower catalog manager: success
I20260929 01:51:35.417615 8790 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:51:35.433957 8710 master_runner.cc:187] Error getting master registration for 127.0.71.125:37929: Network error: Client connection negotiation failed: client connection to 127.0.71.125:37929: connect: Connection refused (error 111),
I20260929 01:51:35.439953 8710 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:51:35.699903 8777 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:51:35.700201 8777 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:51:35.700353 8777 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:51:35.708895 8777 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:51:35.708985 8777 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:51:35.709069 8777 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:51:35.709139 8777 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:51:35.719247 8777 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:37929
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:51:35.720913 8777 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:51:35.722688 8777 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:51:35.730430 8799 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:35.730635 8798 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:35.731458 8801 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:35.732724 8777 server_base.cc:1061] running on GCE node
I20260929 01:51:35.733450 8777 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:51:35.733613 8777 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:51:35.733721 8777 hybrid_clock.cc:648] HybridClock initialized: now 1790646695733685 us; error 0 us; skew 500 ppm
I20260929 01:51:35.734122 8777 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:51:35.736881 8777 webserver.cc:533] Webserver started at http://127.0.71.125:33947/ using document root <none> and password file <none>
I20260929 01:51:35.737597 8777 fs_manager.cc:362] Metadata directory not provided
I20260929 01:51:35.737739 8777 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:51:35.743525 8777 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 01:51:35.746407 8806 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:35.747840 8777 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:51:35.748000 8777 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "291f81ab56bd492aa7755aa1bcf1564e"
format_stamp: "Formatted at 2026-09-29 01:51:20 on dist-test-slave-vz23"
I20260929 01:51:35.748775 8777 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:51:35.785339 8777 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:51:35.786275 8777 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:51:35.786582 8777 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:51:35.807451 8777 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:37929
I20260929 01:51:35.807484 8857 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:37929 every 8 connection(s)
I20260929 01:51:35.809092 8777 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:51:35.815975 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 8777
I20260929 01:51:35.816171 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:51:35.816207 8858 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:51:35.817381 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:43723
--webserver_interface=127.0.71.124
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723 with env {}
I20260929 01:51:35.821336 8858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e: Bootstrap starting.
I20260929 01:51:35.839038 8858 log.cc:826] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e: Log is configured to *not* fsync() on all Append() calls
I20260929 01:51:35.847816 8858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:51:35.848403 8858 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e: Bootstrap complete.
I20260929 01:51:35.855162 8858 raft_consensus.cc:359] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:35.855860 8858 raft_consensus.cc:740] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 291f81ab56bd492aa7755aa1bcf1564e, State: Initialized, Role: FOLLOWER
I20260929 01:51:35.856616 8858 consensus_queue.cc:260] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:35.857512 8862 sys_catalog.cc:455] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:35.857811 8862 sys_catalog.cc:458] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:35.858584 8858 sys_catalog.cc:565] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:51:35.868942 8873 catalog_manager.cc:1308] Loaded cluster ID: 026ed1ecf44c40738ac2f6fc1984a6e8
I20260929 01:51:35.869134 8873 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e: loading cluster ID for follower catalog manager: success
I20260929 01:51:35.873382 8873 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e: acquiring CA information for follower catalog manager: success
I20260929 01:51:35.876008 8873 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:51:35.906440 8777 master_runner.cc:187] Error getting master registration for 127.0.71.124:43723: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43723: connect: Connection refused (error 111),
I20260929 01:51:35.911891 8777 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:51:36.181514 8860 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:51:36.181856 8860 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:51:36.182013 8860 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:51:36.190452 8860 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:51:36.190550 8860 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:51:36.190636 8860 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:51:36.190740 8860 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:51:36.201251 8860 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:43723
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:51:36.202916 8860 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:51:36.204780 8860 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:51:36.212823 8882 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:36.212793 8881 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:36.214038 8884 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:36.215330 8860 server_base.cc:1061] running on GCE node
I20260929 01:51:36.216069 8860 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:51:36.216259 8860 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:51:36.216364 8860 hybrid_clock.cc:648] HybridClock initialized: now 1790646696216343 us; error 0 us; skew 500 ppm
I20260929 01:51:36.216805 8860 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:51:36.219477 8860 webserver.cc:533] Webserver started at http://127.0.71.124:33865/ using document root <none> and password file <none>
I20260929 01:51:36.220289 8860 fs_manager.cc:362] Metadata directory not provided
I20260929 01:51:36.220418 8860 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:51:36.226053 8860 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260929 01:51:36.229135 8889 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:36.230605 8860 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:51:36.230822 8860 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "d181618b1fb340f4aac54452bd2726d3"
format_stamp: "Formatted at 2026-09-29 01:51:20 on dist-test-slave-vz23"
I20260929 01:51:36.231566 8860 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:51:36.248757 8860 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:51:36.249719 8860 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:51:36.250056 8860 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:51:36.272663 8860 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:43723
I20260929 01:51:36.272688 8940 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:43723 every 8 connection(s)
I20260929 01:51:36.274468 8860 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:51:36.281765 8941 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:51:36.282136 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 8860
I20260929 01:51:36.282356 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:51:36.286513 8941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3: Bootstrap starting.
I20260929 01:51:36.305975 8941 log.cc:826] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3: Log is configured to *not* fsync() on all Append() calls
I20260929 01:51:36.314628 8941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:51:36.315337 8941 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3: Bootstrap complete.
I20260929 01:51:36.322217 8941 raft_consensus.cc:359] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:36.322882 8941 raft_consensus.cc:740] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d181618b1fb340f4aac54452bd2726d3, State: Initialized, Role: FOLLOWER
I20260929 01:51:36.323565 8941 consensus_queue.cc:260] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:36.324343 8946 sys_catalog.cc:455] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:36.324699 8946 sys_catalog.cc:458] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:36.325475 8941 sys_catalog.cc:565] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:51:36.337769 8957 catalog_manager.cc:1308] Loaded cluster ID: 026ed1ecf44c40738ac2f6fc1984a6e8
I20260929 01:51:36.337931 8957 catalog_manager.cc:1601] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3: loading cluster ID for follower catalog manager: success
I20260929 01:51:36.342070 8957 catalog_manager.cc:1623] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3: acquiring CA information for follower catalog manager: success
I20260929 01:51:36.353605 8957 catalog_manager.cc:1651] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:51:36.366613 8860 master_runner.cc:264] No leader master found from master d181618b1fb340f4aac54452bd2726d3
I20260929 01:51:36.366770 8860 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:51:36.686100 8960 raft_consensus.cc:493] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:51:36.686422 8960 raft_consensus.cc:515] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:36.688239 8960 leader_election.cc:290] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 291f81ab56bd492aa7755aa1bcf1564e (127.0.71.125:37929), d181618b1fb340f4aac54452bd2726d3 (127.0.71.124:43723)
I20260929 01:51:36.696224 8833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "966a42cbe38040d6aed85c9aa2aec215" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "291f81ab56bd492aa7755aa1bcf1564e" is_pre_election: true
I20260929 01:51:36.696838 8833 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 966a42cbe38040d6aed85c9aa2aec215 in term 2.
I20260929 01:51:36.697414 8916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "966a42cbe38040d6aed85c9aa2aec215" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d181618b1fb340f4aac54452bd2726d3" is_pre_election: true
I20260929 01:51:36.697717 8725 leader_election.cc:304] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 291f81ab56bd492aa7755aa1bcf1564e, 966a42cbe38040d6aed85c9aa2aec215; no voters:
I20260929 01:51:36.698007 8916 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 966a42cbe38040d6aed85c9aa2aec215 in term 2.
I20260929 01:51:36.698084 8960 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 01:51:36.698233 8960 raft_consensus.cc:493] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:51:36.698323 8960 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:51:36.703878 8960 raft_consensus.cc:515] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:36.704780 8960 leader_election.cc:290] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [CANDIDATE]: Term 3 election: Requested vote from peers 291f81ab56bd492aa7755aa1bcf1564e (127.0.71.125:37929), d181618b1fb340f4aac54452bd2726d3 (127.0.71.124:43723)
I20260929 01:51:36.705432 8833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "966a42cbe38040d6aed85c9aa2aec215" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "291f81ab56bd492aa7755aa1bcf1564e"
I20260929 01:51:36.705636 8833 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:51:36.706096 8916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "966a42cbe38040d6aed85c9aa2aec215" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d181618b1fb340f4aac54452bd2726d3"
I20260929 01:51:36.706288 8916 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:51:36.709359 8833 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 966a42cbe38040d6aed85c9aa2aec215 in term 3.
I20260929 01:51:36.709828 8725 leader_election.cc:304] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 291f81ab56bd492aa7755aa1bcf1564e, 966a42cbe38040d6aed85c9aa2aec215; no voters:
I20260929 01:51:36.710144 8960 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 01:51:36.710632 8960 raft_consensus.cc:697] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 3 LEADER]: Becoming Leader. State: Replica: 966a42cbe38040d6aed85c9aa2aec215, State: Running, Role: LEADER
I20260929 01:51:36.711128 8960 consensus_queue.cc:237] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:36.711282 8916 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 966a42cbe38040d6aed85c9aa2aec215 in term 3.
I20260929 01:51:36.713713 8963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 966a42cbe38040d6aed85c9aa2aec215. Latest consensus state: current_term: 3 leader_uuid: "966a42cbe38040d6aed85c9aa2aec215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:36.714020 8963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: This master's current role is: LEADER
I20260929 01:51:36.714633 8965 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:51:36.716773 8965 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:51:36.717522 8965 catalog_manager.cc:1308] Loaded cluster ID: 026ed1ecf44c40738ac2f6fc1984a6e8
I20260929 01:51:36.717624 8965 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:51:36.718704 8965 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:51:36.719452 8965 catalog_manager.cc:6230] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: Loaded TSK: 0
I20260929 01:51:36.720284 8965 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:51:36.738363 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--time_source=system_unsync with env {}
W20260929 01:51:37.086467 8966 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:51:37.086855 8966 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:51:37.086999 8966 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:51:37.095760 8966 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:51:37.096020 8966 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:51:37.106585 8966 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:51:37.108296 8966 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:51:37.110203 8966 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:51:37.120998 8972 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:37.121209 8971 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:37.121601 8974 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:37.122494 8966 server_base.cc:1061] running on GCE node
I20260929 01:51:37.123064 8966 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:51:37.123214 8966 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:51:37.123272 8966 hybrid_clock.cc:648] HybridClock initialized: now 1790646697123256 us; error 0 us; skew 500 ppm
I20260929 01:51:37.123526 8966 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:51:37.125988 8966 webserver.cc:533] Webserver started at http://127.0.71.65:40751/ using document root <none> and password file <none>
I20260929 01:51:37.126807 8966 fs_manager.cc:362] Metadata directory not provided
I20260929 01:51:37.126924 8966 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:51:37.132814 8966 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:51:37.135546 8979 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:37.137171 8966 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:51:37.137362 8966 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "872fd31528444a2986ca624d0c8fa56e"
format_stamp: "Formatted at 2026-09-29 01:51:21 on dist-test-slave-vz23"
I20260929 01:51:37.138206 8966 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:51:37.151575 8966 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:51:37.152406 8966 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:51:37.152745 8966 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:51:37.153666 8966 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:51:37.155391 8966 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:51:37.155462 8966 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:37.155606 8966 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:51:37.155689 8966 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:37.190495 8966 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:42271
I20260929 01:51:37.190531 9091 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:42271 every 8 connection(s)
I20260929 01:51:37.192024 8966 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:51:37.199836 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 8966
I20260929 01:51:37.200052 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
I20260929 01:51:37.205798 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.66:0
--local_ip_for_outbound_sockets=127.0.71.66
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--time_source=system_unsync with env {}
I20260929 01:51:37.211472 9094 heartbeater.cc:344] Connected to a master server at 127.0.71.125:37929
I20260929 01:51:37.211908 9094 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:37.212559 9093 heartbeater.cc:344] Connected to a master server at 127.0.71.126:39309
I20260929 01:51:37.212692 9093 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:37.212872 9092 heartbeater.cc:344] Connected to a master server at 127.0.71.124:43723
I20260929 01:51:37.213009 9092 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:37.213122 9094 heartbeater.cc:507] Master 127.0.71.125:37929 requested a full tablet report, sending...
I20260929 01:51:37.213291 9093 heartbeater.cc:507] Master 127.0.71.126:39309 requested a full tablet report, sending...
I20260929 01:51:37.213650 9092 heartbeater.cc:507] Master 127.0.71.124:43723 requested a full tablet report, sending...
I20260929 01:51:37.215492 8740 ts_manager.cc:194] Registered new tserver with Master: 872fd31528444a2986ca624d0c8fa56e (127.0.71.65:42271)
I20260929 01:51:37.216682 8823 ts_manager.cc:194] Registered new tserver with Master: 872fd31528444a2986ca624d0c8fa56e (127.0.71.65:42271)
I20260929 01:51:37.217469 8740 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:36533
I20260929 01:51:37.218400 8906 ts_manager.cc:194] Registered new tserver with Master: 872fd31528444a2986ca624d0c8fa56e (127.0.71.65:42271)
I20260929 01:51:37.237803 8916 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [term 3 FOLLOWER]: Refusing update from remote peer 966a42cbe38040d6aed85c9aa2aec215: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 01:51:37.238811 9103 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:51:37.243222 9104 sys_catalog.cc:455] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 966a42cbe38040d6aed85c9aa2aec215. Latest consensus state: current_term: 3 leader_uuid: "966a42cbe38040d6aed85c9aa2aec215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:37.243525 9104 sys_catalog.cc:458] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:37.246382 9103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "966a42cbe38040d6aed85c9aa2aec215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:37.246784 9103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: This master's current role is: LEADER
I20260929 01:51:37.247124 8833 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [term 3 FOLLOWER]: Refusing update from remote peer 966a42cbe38040d6aed85c9aa2aec215: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 01:51:37.259893 9106 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [LEADER]: Connected to new peer: Peer: permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:51:37.260677 9104 sys_catalog.cc:455] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "966a42cbe38040d6aed85c9aa2aec215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:37.260907 9104 sys_catalog.cc:458] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:37.265363 9114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "966a42cbe38040d6aed85c9aa2aec215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:37.265820 9114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:37.266248 9111 sys_catalog.cc:455] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 966a42cbe38040d6aed85c9aa2aec215. Latest consensus state: current_term: 3 leader_uuid: "966a42cbe38040d6aed85c9aa2aec215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:37.266453 9111 sys_catalog.cc:458] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:51:37.566257 9102 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:51:37.566653 9102 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:51:37.566800 9102 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:51:37.575487 9102 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:51:37.575749 9102 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.66
I20260929 01:51:37.585789 9102 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:51:37.587456 9102 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:51:37.589408 9102 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:51:37.601292 9122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:37.601208 9120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:37.601228 9119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:37.602706 9102 server_base.cc:1061] running on GCE node
I20260929 01:51:37.603343 9102 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:51:37.603534 9102 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:51:37.603636 9102 hybrid_clock.cc:648] HybridClock initialized: now 1790646697603609 us; error 0 us; skew 500 ppm
I20260929 01:51:37.603977 9102 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:51:37.606338 9102 webserver.cc:533] Webserver started at http://127.0.71.66:35281/ using document root <none> and password file <none>
I20260929 01:51:37.607110 9102 fs_manager.cc:362] Metadata directory not provided
I20260929 01:51:37.607273 9102 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:51:37.613005 9102 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 01:51:37.615681 9127 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:37.617193 9102 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:51:37.617374 9102 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/wal
uuid: "f4251826e5f34caf8ff26cfb99beb84e"
format_stamp: "Formatted at 2026-09-29 01:51:21 on dist-test-slave-vz23"
I20260929 01:51:37.618132 9102 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:51:37.633796 9102 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:51:37.634667 9102 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:51:37.634995 9102 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:51:37.635953 9102 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:51:37.637672 9102 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:51:37.637750 9102 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:37.637888 9102 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:51:37.637943 9102 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:37.676059 9102 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.66:44921
I20260929 01:51:37.676092 9239 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.66:44921 every 8 connection(s)
I20260929 01:51:37.677774 9102 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
I20260929 01:51:37.680863 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 9102
I20260929 01:51:37.681061 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-1/wal/instance
I20260929 01:51:37.688374 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.67:0
--local_ip_for_outbound_sockets=127.0.71.67
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--time_source=system_unsync with env {}
I20260929 01:51:37.698060 9242 heartbeater.cc:344] Connected to a master server at 127.0.71.125:37929
I20260929 01:51:37.698472 9242 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:37.698609 9241 heartbeater.cc:344] Connected to a master server at 127.0.71.126:39309
I20260929 01:51:37.698819 9241 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:37.699776 9241 heartbeater.cc:507] Master 127.0.71.126:39309 requested a full tablet report, sending...
I20260929 01:51:37.700066 9242 heartbeater.cc:507] Master 127.0.71.125:37929 requested a full tablet report, sending...
I20260929 01:51:37.701515 8740 ts_manager.cc:194] Registered new tserver with Master: f4251826e5f34caf8ff26cfb99beb84e (127.0.71.66:44921)
I20260929 01:51:37.701597 8823 ts_manager.cc:194] Registered new tserver with Master: f4251826e5f34caf8ff26cfb99beb84e (127.0.71.66:44921)
I20260929 01:51:37.702375 9240 heartbeater.cc:344] Connected to a master server at 127.0.71.124:43723
I20260929 01:51:37.702488 9240 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:37.702466 8740 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.66:37645
I20260929 01:51:37.703045 9240 heartbeater.cc:507] Master 127.0.71.124:43723 requested a full tablet report, sending...
I20260929 01:51:37.704222 8906 ts_manager.cc:194] Registered new tserver with Master: f4251826e5f34caf8ff26cfb99beb84e (127.0.71.66:44921)
I20260929 01:51:37.778445 9106 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "966a42cbe38040d6aed85c9aa2aec215" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:37.778784 9106 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: This master's current role is: LEADER
W20260929 01:51:38.032388 9247 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:51:38.032829 9247 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:51:38.032969 9247 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:51:38.041502 9247 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:51:38.041664 9247 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.67
I20260929 01:51:38.051730 9247 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:51:38.053325 9247 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:51:38.055210 9247 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:51:38.065078 9255 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:38.065079 9256 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:38.066525 9247 server_base.cc:1061] running on GCE node
W20260929 01:51:38.066566 9258 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:38.067334 9247 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:51:38.067481 9247 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:51:38.067545 9247 hybrid_clock.cc:648] HybridClock initialized: now 1790646698067529 us; error 0 us; skew 500 ppm
I20260929 01:51:38.067798 9247 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:51:38.070081 9247 webserver.cc:533] Webserver started at http://127.0.71.67:39303/ using document root <none> and password file <none>
I20260929 01:51:38.070777 9247 fs_manager.cc:362] Metadata directory not provided
I20260929 01:51:38.070878 9247 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:51:38.077139 9247 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 01:51:38.079869 9263 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:38.081492 9247 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:51:38.081729 9247 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/wal
uuid: "d1e530562ffb4eff970aca2d4bed29c1"
format_stamp: "Formatted at 2026-09-29 01:51:22 on dist-test-slave-vz23"
I20260929 01:51:38.082508 9247 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:51:38.116045 9247 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:51:38.117103 9247 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:51:38.117455 9247 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:51:38.118484 9247 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:51:38.120306 9247 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:51:38.120419 9247 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:38.120602 9247 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:51:38.120685 9247 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:38.158958 9247 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.67:40019
I20260929 01:51:38.158989 9375 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.67:40019 every 8 connection(s)
I20260929 01:51:38.160652 9247 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
I20260929 01:51:38.165650 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 9247
I20260929 01:51:38.165844 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-2/wal/instance
I20260929 01:51:38.178701 9377 heartbeater.cc:344] Connected to a master server at 127.0.71.126:39309
I20260929 01:51:38.178784 9376 heartbeater.cc:344] Connected to a master server at 127.0.71.124:43723
I20260929 01:51:38.178812 9378 heartbeater.cc:344] Connected to a master server at 127.0.71.125:37929
I20260929 01:51:38.179170 9378 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:38.179173 9377 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:38.179175 9376 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:38.180166 9376 heartbeater.cc:507] Master 127.0.71.124:43723 requested a full tablet report, sending...
I20260929 01:51:38.180208 9377 heartbeater.cc:507] Master 127.0.71.126:39309 requested a full tablet report, sending...
I20260929 01:51:38.180243 9378 heartbeater.cc:507] Master 127.0.71.125:37929 requested a full tablet report, sending...
I20260929 01:51:38.181958 8740 ts_manager.cc:194] Registered new tserver with Master: d1e530562ffb4eff970aca2d4bed29c1 (127.0.71.67:40019)
I20260929 01:51:38.182200 8906 ts_manager.cc:194] Registered new tserver with Master: d1e530562ffb4eff970aca2d4bed29c1 (127.0.71.67:40019)
I20260929 01:51:38.182228 8823 ts_manager.cc:194] Registered new tserver with Master: d1e530562ffb4eff970aca2d4bed29c1 (127.0.71.67:40019)
I20260929 01:51:38.183040 8740 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.67:58843
I20260929 01:51:38.191352 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:51:38.198964 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:51:38.199165 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 8710
I20260929 01:51:38.200071 8770 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:51:38.219913 9093 heartbeater.cc:499] Master 127.0.71.126:39309 was elected leader, sending a full tablet report...
I20260929 01:51:38.314074 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 8710
W20260929 01:51:38.329929 9093 heartbeater.cc:646] Failed to heartbeat to 127.0.71.126:39309 (0 consecutive failures): Network error: Failed to send heartbeat to master: recv got EOF from 127.0.71.126:39309 (error 108)
I20260929 01:51:38.330336 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:39309
--webserver_interface=127.0.71.126
--webserver_port=36047
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723 with env {}
W20260929 01:51:38.661494 9384 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:51:38.661863 9384 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:51:38.661933 9384 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:51:38.670194 9384 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:51:38.670279 9384 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:51:38.670336 9384 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:51:38.670377 9384 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:51:38.680769 9384 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:39309
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=36047
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:51:38.682236 9384 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:51:38.683905 9384 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:51:38.693203 9389 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:38.693234 9390 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:38.693266 9392 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:38.693786 9384 server_base.cc:1061] running on GCE node
I20260929 01:51:38.694713 9384 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:51:38.694895 9384 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:51:38.695001 9384 hybrid_clock.cc:648] HybridClock initialized: now 1790646698694965 us; error 0 us; skew 500 ppm
I20260929 01:51:38.695325 9384 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:51:38.698109 9384 webserver.cc:533] Webserver started at http://127.0.71.126:36047/ using document root <none> and password file <none>
I20260929 01:51:38.698827 9384 fs_manager.cc:362] Metadata directory not provided
I20260929 01:51:38.698971 9384 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:51:38.704780 9241 heartbeater.cc:499] Master 127.0.71.126:39309 was elected leader, sending a full tablet report...
I20260929 01:51:38.704854 9384 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
W20260929 01:51:38.706730 9241 heartbeater.cc:646] Failed to heartbeat to 127.0.71.126:39309 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.71.126:39309: connect: Connection refused (error 111)
I20260929 01:51:38.708057 9398 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:38.709467 9384 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:51:38.709666 9384 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "966a42cbe38040d6aed85c9aa2aec215"
format_stamp: "Formatted at 2026-09-29 01:51:19 on dist-test-slave-vz23"
I20260929 01:51:38.710438 9384 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:51:38.728749 9384 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:51:38.729734 9384 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:51:38.730041 9384 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:51:38.751502 9384 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:39309
I20260929 01:51:38.751559 9449 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:39309 every 8 connection(s)
I20260929 01:51:38.753268 9384 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:51:38.759748 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 9384
I20260929 01:51:38.760089 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 8966
I20260929 01:51:38.760776 9450 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:51:38.765072 9450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: Bootstrap starting.
I20260929 01:51:38.770241 9087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:51:38.783707 9450 log.cc:826] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: Log is configured to *not* fsync() on all Append() calls
I20260929 01:51:38.792937 9450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:51:38.793481 9450 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: Bootstrap complete.
I20260929 01:51:38.800204 9450 raft_consensus.cc:359] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:38.800973 9450 raft_consensus.cc:740] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 966a42cbe38040d6aed85c9aa2aec215, State: Initialized, Role: FOLLOWER
I20260929 01:51:38.801780 9450 consensus_queue.cc:260] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:38.802634 9456 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:38.802953 9456 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:38.803584 9450 sys_catalog.cc:565] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:51:38.826057 9467 catalog_manager.cc:1308] Loaded cluster ID: 026ed1ecf44c40738ac2f6fc1984a6e8
I20260929 01:51:38.826256 9467 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: loading cluster ID for follower catalog manager: success
I20260929 01:51:38.830519 9467 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: acquiring CA information for follower catalog manager: success
I20260929 01:51:38.833316 9467 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:51:38.870916 9384 master_runner.cc:264] No leader master found from master 966a42cbe38040d6aed85c9aa2aec215
I20260929 01:51:38.871102 9384 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:51:38.894739 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 8966
I20260929 01:51:38.912343 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:42271
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--webserver_port=40751
--webserver_interface=127.0.71.65
--time_source=system_unsync with env {}
I20260929 01:51:39.185479 9377 heartbeater.cc:499] Master 127.0.71.126:39309 was elected leader, sending a full tablet report...
I20260929 01:51:39.197018 9415 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d1e530562ffb4eff970aca2d4bed29c1" instance_seqno: 1790646698149625) as {username='slave'} at 127.0.71.67:41099; Asking this server to re-register.
I20260929 01:51:39.198066 9377 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:39.200057 9415 ts_manager.cc:194] Registered new tserver with Master: d1e530562ffb4eff970aca2d4bed29c1 (127.0.71.67:40019)
W20260929 01:51:39.274958 9473 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:51:39.275379 9473 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:51:39.275519 9473 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260929 01:51:39.280429 9478 raft_consensus.cc:493] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 966a42cbe38040d6aed85c9aa2aec215)
I20260929 01:51:39.280699 9478 raft_consensus.cc:515] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:39.282441 9478 leader_election.cc:290] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 966a42cbe38040d6aed85c9aa2aec215 (127.0.71.126:39309), d181618b1fb340f4aac54452bd2726d3 (127.0.71.124:43723)
W20260929 01:51:39.285101 9473 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:51:39.285343 9473 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:51:39.290971 8916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "291f81ab56bd492aa7755aa1bcf1564e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d181618b1fb340f4aac54452bd2726d3" is_pre_election: true
I20260929 01:51:39.291323 9425 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "291f81ab56bd492aa7755aa1bcf1564e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "966a42cbe38040d6aed85c9aa2aec215" is_pre_election: true
I20260929 01:51:39.291908 9425 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 291f81ab56bd492aa7755aa1bcf1564e in term 3.
I20260929 01:51:39.292629 8807 leader_election.cc:304] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 291f81ab56bd492aa7755aa1bcf1564e, 966a42cbe38040d6aed85c9aa2aec215; no voters: d181618b1fb340f4aac54452bd2726d3
I20260929 01:51:39.293061 9478 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 01:51:39.293216 9478 raft_consensus.cc:493] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [term 3 FOLLOWER]: Starting leader election (detected failure of leader 966a42cbe38040d6aed85c9aa2aec215)
I20260929 01:51:39.293365 9478 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [term 3 FOLLOWER]: Advancing to term 4
I20260929 01:51:39.295619 9478 raft_consensus.cc:515] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:39.296300 9478 leader_election.cc:290] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [CANDIDATE]: Term 4 election: Requested vote from peers 966a42cbe38040d6aed85c9aa2aec215 (127.0.71.126:39309), d181618b1fb340f4aac54452bd2726d3 (127.0.71.124:43723)
I20260929 01:51:39.296818 9425 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "291f81ab56bd492aa7755aa1bcf1564e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "966a42cbe38040d6aed85c9aa2aec215"
I20260929 01:51:39.296919 8916 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "291f81ab56bd492aa7755aa1bcf1564e" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d181618b1fb340f4aac54452bd2726d3"
I20260929 01:51:39.297030 9425 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 3 FOLLOWER]: Advancing to term 4
I20260929 01:51:39.298029 9473 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:42271
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=40751
--tserver_master_addrs=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:51:39.299569 9473 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:51:39.300933 9425 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 291f81ab56bd492aa7755aa1bcf1564e in term 4.
I20260929 01:51:39.301342 8807 leader_election.cc:304] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 291f81ab56bd492aa7755aa1bcf1564e, 966a42cbe38040d6aed85c9aa2aec215; no voters: d181618b1fb340f4aac54452bd2726d3
I20260929 01:51:39.301569 9473 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260929 01:51:39.301664 9478 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [term 4 FOLLOWER]: Leader election won for term 4
I20260929 01:51:39.302210 9478 raft_consensus.cc:697] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [term 4 LEADER]: Becoming Leader. State: Replica: 291f81ab56bd492aa7755aa1bcf1564e, State: Running, Role: LEADER
I20260929 01:51:39.302640 9478 consensus_queue.cc:237] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:39.304418 9481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 291f81ab56bd492aa7755aa1bcf1564e. Latest consensus state: current_term: 4 leader_uuid: "291f81ab56bd492aa7755aa1bcf1564e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:39.304757 9481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [sys.catalog]: This master's current role is: LEADER
I20260929 01:51:39.305522 9483 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:51:39.307508 9483 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:51:39.308280 9483 catalog_manager.cc:1308] Loaded cluster ID: 026ed1ecf44c40738ac2f6fc1984a6e8
I20260929 01:51:39.308355 9483 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:51:39.309723 9483 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:51:39.310523 9483 catalog_manager.cc:6230] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e: Loaded TSK: 0
I20260929 01:51:39.311743 9483 catalog_manager.cc:1563] Initializing in-progress tserver states...
W20260929 01:51:39.311954 9485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:39.312615 9486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:39.314138 9488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:39.315016 9473 server_base.cc:1061] running on GCE node
I20260929 01:51:39.315698 9473 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:51:39.315876 9473 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:51:39.315979 9473 hybrid_clock.cc:648] HybridClock initialized: now 1790646699315939 us; error 0 us; skew 500 ppm
I20260929 01:51:39.316283 9473 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:51:39.318523 9473 webserver.cc:533] Webserver started at http://127.0.71.65:40751/ using document root <none> and password file <none>
I20260929 01:51:39.319276 9473 fs_manager.cc:362] Metadata directory not provided
I20260929 01:51:39.319401 9473 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:51:39.325002 9473 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:51:39.327704 9493 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:39.329154 9473 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:51:39.329327 9473 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "872fd31528444a2986ca624d0c8fa56e"
format_stamp: "Formatted at 2026-09-29 01:51:21 on dist-test-slave-vz23"
I20260929 01:51:39.330049 9473 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:51:39.348759 9473 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:51:39.349633 9473 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:51:39.349920 9473 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:51:39.350816 9473 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:51:39.352418 9473 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:51:39.352573 9473 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 01:51:39.352728 9473 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:51:39.352798 9473 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:39.388026 9473 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:42271
I20260929 01:51:39.388072 9605 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:42271 every 8 connection(s)
I20260929 01:51:39.389616 9473 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:51:39.397879 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 9473
I20260929 01:51:39.417234 9606 heartbeater.cc:344] Connected to a master server at 127.0.71.124:43723
I20260929 01:51:39.417467 9607 heartbeater.cc:344] Connected to a master server at 127.0.71.126:39309
I20260929 01:51:39.417655 9607 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:39.417662 9606 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:39.417840 9608 heartbeater.cc:344] Connected to a master server at 127.0.71.125:37929
I20260929 01:51:39.417968 9608 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:39.418871 9606 heartbeater.cc:507] Master 127.0.71.124:43723 requested a full tablet report, sending...
I20260929 01:51:39.419297 9607 heartbeater.cc:507] Master 127.0.71.126:39309 requested a full tablet report, sending...
I20260929 01:51:39.420415 9608 heartbeater.cc:507] Master 127.0.71.125:37929 requested a full tablet report, sending...
I20260929 01:51:39.420735 9415 ts_manager.cc:194] Registered new tserver with Master: 872fd31528444a2986ca624d0c8fa56e (127.0.71.65:42271)
I20260929 01:51:39.421660 8823 ts_manager.cc:194] Re-registered known tserver with Master: 872fd31528444a2986ca624d0c8fa56e (127.0.71.65:42271)
I20260929 01:51:39.421766 8906 ts_manager.cc:194] Re-registered known tserver with Master: 872fd31528444a2986ca624d0c8fa56e (127.0.71.65:42271)
I20260929 01:51:39.423960 8823 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:51253
I20260929 01:51:39.515094 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 9473
I20260929 01:51:39.536284 9601 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:51:39.567488 9619 raft_consensus.cc:493] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 966a42cbe38040d6aed85c9aa2aec215)
I20260929 01:51:39.567793 9619 raft_consensus.cc:515] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } }
I20260929 01:51:39.569761 9619 leader_election.cc:290] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 966a42cbe38040d6aed85c9aa2aec215 (127.0.71.126:39309), 291f81ab56bd492aa7755aa1bcf1564e (127.0.71.125:37929)
I20260929 01:51:39.570467 8833 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d181618b1fb340f4aac54452bd2726d3" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "291f81ab56bd492aa7755aa1bcf1564e" is_pre_election: true
I20260929 01:51:39.578011 9425 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d181618b1fb340f4aac54452bd2726d3" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "966a42cbe38040d6aed85c9aa2aec215" is_pre_election: true
I20260929 01:51:39.578395 9425 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d181618b1fb340f4aac54452bd2726d3 in current term 4: Already voted for candidate 291f81ab56bd492aa7755aa1bcf1564e in this term.
I20260929 01:51:39.579154 8890 leader_election.cc:304] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d181618b1fb340f4aac54452bd2726d3; no voters: 291f81ab56bd492aa7755aa1bcf1564e, 966a42cbe38040d6aed85c9aa2aec215
I20260929 01:51:39.579641 9619 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [term 3 FOLLOWER]: Advancing to term 4
I20260929 01:51:39.582777 9619 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
I20260929 01:51:39.642184 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 9473
I20260929 01:51:39.659291 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:42271
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--webserver_port=40751
--webserver_interface=127.0.71.65
--time_source=system_unsync
--fault_before_start=1.0 with env {}
I20260929 01:51:39.701431 9425 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [term 4 FOLLOWER]: Refusing update from remote peer 291f81ab56bd492aa7755aa1bcf1564e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260929 01:51:39.702322 9481 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [LEADER]: Connected to new peer: Peer: permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 01:51:39.705973 9622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 291f81ab56bd492aa7755aa1bcf1564e. Latest consensus state: current_term: 4 leader_uuid: "291f81ab56bd492aa7755aa1bcf1564e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:39.706275 9622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:39.717702 8916 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [term 4 FOLLOWER]: Refusing update from remote peer 291f81ab56bd492aa7755aa1bcf1564e: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260929 01:51:39.718717 9478 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [LEADER]: Connected to new peer: Peer: permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 01:51:39.722692 9622 sys_catalog.cc:455] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "291f81ab56bd492aa7755aa1bcf1564e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:39.723021 9622 sys_catalog.cc:458] T 00000000000000000000000000000000 P 966a42cbe38040d6aed85c9aa2aec215 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:39.724637 9478 sys_catalog.cc:455] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "291f81ab56bd492aa7755aa1bcf1564e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:39.724879 9478 sys_catalog.cc:458] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [sys.catalog]: This master's current role is: LEADER
I20260929 01:51:39.726153 9619 sys_catalog.cc:455] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 291f81ab56bd492aa7755aa1bcf1564e. Latest consensus state: current_term: 4 leader_uuid: "291f81ab56bd492aa7755aa1bcf1564e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:39.726147 9629 sys_catalog.cc:455] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "291f81ab56bd492aa7755aa1bcf1564e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:39.726439 9619 sys_catalog.cc:458] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:39.726452 9629 sys_catalog.cc:458] T 00000000000000000000000000000000 P d181618b1fb340f4aac54452bd2726d3 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:39.728330 9241 heartbeater.cc:344] Connected to a master server at 127.0.71.126:39309
I20260929 01:51:39.728572 9241 heartbeater.cc:499] Master 127.0.71.126:39309 was elected leader, sending a full tablet report...
I20260929 01:51:39.729573 9415 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f4251826e5f34caf8ff26cfb99beb84e" instance_seqno: 1790646697667111) as {username='slave'} at 127.0.71.66:38663; Asking this server to re-register.
I20260929 01:51:39.730307 9241 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:39.731545 9415 ts_manager.cc:194] Registered new tserver with Master: f4251826e5f34caf8ff26cfb99beb84e (127.0.71.66:44921)
W20260929 01:51:40.002346 9621 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:51:40.002663 9621 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:51:40.002758 9621 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:51:40.002818 9621 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:51:40.011720 9621 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:51:40.011917 9621 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:51:40.021838 9621 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:42271
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=40751
--tserver_master_addrs=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:51:40.023414 9621 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:51:40.025287 9621 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:51:40.035141 9635 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:40.035172 9637 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:40.035808 9634 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:40.035923 9621 server_base.cc:1061] running on GCE node
I20260929 01:51:40.036669 9621 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:51:40.036876 9621 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:51:40.037001 9621 hybrid_clock.cc:648] HybridClock initialized: now 1790646700036963 us; error 0 us; skew 500 ppm
I20260929 01:51:40.037398 9621 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:51:40.039541 9621 webserver.cc:533] Webserver started at http://127.0.71.65:40751/ using document root <none> and password file <none>
I20260929 01:51:40.040216 9621 fs_manager.cc:362] Metadata directory not provided
I20260929 01:51:40.040314 9621 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:51:40.046012 9621 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 01:51:40.048658 9642 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:40.050120 9621 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:51:40.050302 9621 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "872fd31528444a2986ca624d0c8fa56e"
format_stamp: "Formatted at 2026-09-29 01:51:21 on dist-test-slave-vz23"
I20260929 01:51:40.051029 9621 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:51:40.076032 9621 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:51:40.077214 9621 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:51:40.077536 9621 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:51:40.078599 9621 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:51:40.080500 9621 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:51:40.080633 9621 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 01:51:40.080783 9621 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:51:40.080871 9621 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:51:40.081247 9621 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:51:40.081336 9621 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:51:40.081452 9621 fault_injection.cc:59] injecting fault for kudu (pid 9621): FLAGS_fault_before_start (process will exit)
I20260929 01:51:40.099076 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:42271
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--webserver_port=40751
--webserver_interface=127.0.71.65
--time_source=system_unsync with env {}
I20260929 01:51:40.246829 9651 sys_catalog.cc:455] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "291f81ab56bd492aa7755aa1bcf1564e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "966a42cbe38040d6aed85c9aa2aec215" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39309 } } peers { permanent_uuid: "291f81ab56bd492aa7755aa1bcf1564e" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 37929 } } peers { permanent_uuid: "d181618b1fb340f4aac54452bd2726d3" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43723 } } }
I20260929 01:51:40.247165 9651 sys_catalog.cc:458] T 00000000000000000000000000000000 P 291f81ab56bd492aa7755aa1bcf1564e [sys.catalog]: This master's current role is: LEADER
W20260929 01:51:40.431056 9650 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:51:40.431449 9650 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:51:40.431617 9650 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:51:40.440402 9650 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:51:40.440687 9650 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:51:40.450851 9650 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:42271
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=40751
--tserver_master_addrs=127.0.71.126:39309,127.0.71.125:37929,127.0.71.124:43723
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:51:40.452633 9650 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:51:40.454810 9650 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:51:40.464787 9659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:40.465163 9662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:40.464870 9660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:40.466572 9650 server_base.cc:1061] running on GCE node
I20260929 01:51:40.467238 9650 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:51:40.467414 9650 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:51:40.467509 9650 hybrid_clock.cc:648] HybridClock initialized: now 1790646700467489 us; error 0 us; skew 500 ppm
I20260929 01:51:40.467830 9650 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:51:40.470216 9650 webserver.cc:533] Webserver started at http://127.0.71.65:40751/ using document root <none> and password file <none>
I20260929 01:51:40.471042 9650 fs_manager.cc:362] Metadata directory not provided
I20260929 01:51:40.471170 9650 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:51:40.477164 9650 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:51:40.479930 9667 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:40.481421 9650 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:51:40.481596 9650 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "872fd31528444a2986ca624d0c8fa56e"
format_stamp: "Formatted at 2026-09-29 01:51:21 on dist-test-slave-vz23"
I20260929 01:51:40.482281 9650 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:51:40.494828 9650 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:51:40.495677 9650 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:51:40.495959 9650 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:51:40.496917 9650 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:51:40.498543 9650 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:51:40.498611 9650 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:40.498766 9650 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:51:40.498827 9650 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:40.534770 9650 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:42271
I20260929 01:51:40.534863 9779 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:42271 every 8 connection(s)
I20260929 01:51:40.536351 9650 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:51:40.538470 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 9650
I20260929 01:51:40.538767 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 9650
I20260929 01:51:40.564375 9780 heartbeater.cc:344] Connected to a master server at 127.0.71.124:43723
I20260929 01:51:40.564653 9775 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:51:40.564874 9780 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:40.650525 9781 heartbeater.cc:344] Connected to a master server at 127.0.71.126:39309
I20260929 01:51:40.650786 9781 heartbeater.cc:461] Registering TS with master...
I20260929 01:51:40.651832 9780 heartbeater.cc:507] Master 127.0.71.124:43723 requested a full tablet report, sending...
I20260929 01:51:40.651893 9781 heartbeater.cc:507] Master 127.0.71.126:39309 requested a full tablet report, sending...
I20260929 01:51:40.653594 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 9650
I20260929 01:51:40.653723 8906 ts_manager.cc:194] Re-registered known tserver with Master: 872fd31528444a2986ca624d0c8fa56e (127.0.71.65:42271)
I20260929 01:51:40.654414 9415 ts_manager.cc:194] Re-registered known tserver with Master: 872fd31528444a2986ca624d0c8fa56e (127.0.71.65:42271)
W20260929 01:51:40.669911 8808 connection.cc:573] server connection from 127.0.71.65:46921 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 01:51:40.669991 8893 connection.cc:573] server connection from 127.0.71.65:54401 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 01:51:40.669991 9399 connection.cc:573] server connection from 127.0.71.65:54073 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:51:40.670492 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 9102
I20260929 01:51:40.678635 9235 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:51:40.763375 9242 heartbeater.cc:499] Master 127.0.71.125:37929 was elected leader, sending a full tablet report...
I20260929 01:51:40.765230 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 9102
I20260929 01:51:40.780947 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 9247
I20260929 01:51:40.789196 9371 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:51:40.882058 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 9247
I20260929 01:51:40.898054 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 9384
I20260929 01:51:40.904872 9445 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:51:41.017900 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 9384
I20260929 01:51:41.037881 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 8777
I20260929 01:51:41.037823 8777 master_runner.cc:187] Error getting master registration for 127.0.71.126:39309: Network error: recv got EOF from 127.0.71.126:39309 (error 108),
I20260929 01:51:41.038061 8777 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:51:41.039108 8853 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:51:41.151873 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 8777
W20260929 01:51:41.171686 8893 connection.cc:573] server connection from 127.0.0.1:37992 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:51:41.172117 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 8860
I20260929 01:51:41.173473 8936 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:51:41.280174 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 8860
Shutting down hive metastore.
[ OK ] ExternalMiniClusterTest.TestBasicOperation/3 (38439 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/4
2026-09-29T01:51:41Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:51:41Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/security.properties
2026-09-29 01:51:49: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260929 01:51:57.543820 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:44069
--webserver_interface=127.0.71.126
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795 with env {}
W20260929 01:51:57.868076 10083 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:51:57.868440 10083 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:51:57.868608 10083 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:51:57.877017 10083 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:51:57.877110 10083 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:51:57.877194 10083 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:51:57.877292 10083 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:51:57.887480 10083 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:44069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:51:57.889218 10083 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:51:57.891021 10083 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:51:57.899408 10088 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:57.899471 10089 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:57.899762 10091 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:57.900432 10083 server_base.cc:1061] running on GCE node
I20260929 01:51:57.901244 10083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:51:57.902567 10083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:51:57.903838 10083 hybrid_clock.cc:648] HybridClock initialized: now 1790646717903796 us; error 56 us; skew 500 ppm
I20260929 01:51:57.904209 10083 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:51:57.906454 10083 webserver.cc:533] Webserver started at http://127.0.71.126:36963/ using document root <none> and password file <none>
I20260929 01:51:57.907202 10083 fs_manager.cc:362] Metadata directory not provided
I20260929 01:51:57.907341 10083 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:51:57.907706 10083 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:51:57.910169 10083 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6"
format_stamp: "Formatted at 2026-09-29 01:51:57 on dist-test-slave-vz23"
I20260929 01:51:57.910960 10083 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6"
format_stamp: "Formatted at 2026-09-29 01:51:57 on dist-test-slave-vz23"
I20260929 01:51:57.916421 10083 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 01:51:57.919876 10097 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:57.921308 10083 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 01:51:57.921480 10083 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6"
format_stamp: "Formatted at 2026-09-29 01:51:57 on dist-test-slave-vz23"
I20260929 01:51:57.921766 10083 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:51:57.939476 10083 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:51:57.940491 10083 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:51:57.940788 10083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:51:57.961009 10083 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:44069
I20260929 01:51:57.961030 10148 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:44069 every 8 connection(s)
I20260929 01:51:57.962711 10083 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:51:57.963487 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 10083
I20260929 01:51:57.963814 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:51:57.964968 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:46041
--webserver_interface=127.0.71.125
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795 with env {}
I20260929 01:51:57.967304 10149 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:51:57.973161 10149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:51:57.989598 10149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:51:57.991041 10101 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.125:46041: connect: Connection refused (error 111)
W20260929 01:51:57.992827 10149 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:46041: Network error: Client connection negotiation failed: client connection to 127.0.71.125:46041: connect: Connection refused (error 111)
I20260929 01:51:58.042235 10149 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } attempt: 1
W20260929 01:51:58.044437 10149 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:46041: Network error: Client connection negotiation failed: client connection to 127.0.71.125:46041: connect: Connection refused (error 111)
I20260929 01:51:58.112759 10149 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } attempt: 2
W20260929 01:51:58.115092 10149 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:46041: Network error: Client connection negotiation failed: client connection to 127.0.71.125:46041: connect: Connection refused (error 111)
I20260929 01:51:58.206403 10149 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } attempt: 3
W20260929 01:51:58.208482 10149 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:46041: Network error: Client connection negotiation failed: client connection to 127.0.71.125:46041: connect: Connection refused (error 111)
W20260929 01:51:58.310546 10151 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:51:58.310920 10151 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:51:58.311077 10151 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:51:58.320161 10151 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:51:58.320261 10151 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:51:58.320355 10151 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:51:58.320425 10151 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:51:58.331264 10151 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:46041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:51:58.333011 10151 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:51:58.334771 10151 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:51:58.342628 10159 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:58.342641 10160 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:58.343411 10162 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:58.344070 10151 server_base.cc:1061] running on GCE node
I20260929 01:51:58.344882 10151 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:51:58.346292 10151 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:51:58.347527 10151 hybrid_clock.cc:648] HybridClock initialized: now 1790646718347468 us; error 55 us; skew 500 ppm
I20260929 01:51:58.347919 10151 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:51:58.350314 10151 webserver.cc:533] Webserver started at http://127.0.71.125:43181/ using document root <none> and password file <none>
I20260929 01:51:58.351063 10151 fs_manager.cc:362] Metadata directory not provided
I20260929 01:51:58.351199 10151 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:51:58.351553 10151 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:51:58.351776 10149 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } attempt: 4
W20260929 01:51:58.353843 10149 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:46041: Network error: Client connection negotiation failed: client connection to 127.0.71.125:46041: connect: Connection refused (error 111)
I20260929 01:51:58.354591 10151 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/data/instance:
uuid: "fd52acb9c5384217a5983864fb4140ab"
format_stamp: "Formatted at 2026-09-29 01:51:58 on dist-test-slave-vz23"
I20260929 01:51:58.355306 10151 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/wal/instance:
uuid: "fd52acb9c5384217a5983864fb4140ab"
format_stamp: "Formatted at 2026-09-29 01:51:58 on dist-test-slave-vz23"
I20260929 01:51:58.360688 10151 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 01:51:58.364174 10168 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:58.365675 10151 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:51:58.365866 10151 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "fd52acb9c5384217a5983864fb4140ab"
format_stamp: "Formatted at 2026-09-29 01:51:58 on dist-test-slave-vz23"
I20260929 01:51:58.366093 10151 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:51:58.390848 10151 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:51:58.391860 10151 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:51:58.392169 10151 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:51:58.414266 10151 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:46041
I20260929 01:51:58.414373 10219 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:46041 every 8 connection(s)
I20260929 01:51:58.415997 10151 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:51:58.420500 10220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:51:58.424793 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 10151
I20260929 01:51:58.425004 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:51:58.426023 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:42795
--webserver_interface=127.0.71.124
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795 with env {}
I20260929 01:51:58.426456 10220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:51:58.441290 10220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:51:58.452054 10220 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:51:58.453413 10172 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.124:42795: connect: Connection refused (error 111)
W20260929 01:51:58.455186 10220 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:42795: Network error: Client connection negotiation failed: client connection to 127.0.71.124:42795: connect: Connection refused (error 111)
I20260929 01:51:58.504573 10220 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } attempt: 1
W20260929 01:51:58.506815 10220 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:42795: Network error: Client connection negotiation failed: client connection to 127.0.71.124:42795: connect: Connection refused (error 111)
I20260929 01:51:58.575124 10220 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } attempt: 2
W20260929 01:51:58.577294 10220 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:42795: Network error: Client connection negotiation failed: client connection to 127.0.71.124:42795: connect: Connection refused (error 111)
I20260929 01:51:58.653115 10149 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } attempt: 5
I20260929 01:51:58.660087 10149 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:51:58.662206 10149 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:42795: Network error: Client connection negotiation failed: client connection to 127.0.71.124:42795: connect: Connection refused (error 111)
I20260929 01:51:58.668568 10220 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } attempt: 3
W20260929 01:51:58.670678 10220 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:42795: Network error: Client connection negotiation failed: client connection to 127.0.71.124:42795: connect: Connection refused (error 111)
I20260929 01:51:58.713472 10149 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } attempt: 1
W20260929 01:51:58.715858 10149 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:42795: Network error: Client connection negotiation failed: client connection to 127.0.71.124:42795: connect: Connection refused (error 111)
W20260929 01:51:58.771903 10223 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:51:58.772220 10223 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:51:58.772286 10223 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:51:58.780793 10223 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:51:58.780890 10223 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:51:58.780944 10223 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:51:58.780984 10223 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:51:58.784137 10149 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } attempt: 2
W20260929 01:51:58.786317 10149 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:42795: Network error: Client connection negotiation failed: client connection to 127.0.71.124:42795: connect: Connection refused (error 111)
I20260929 01:51:58.791662 10223 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:42795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:51:58.793290 10223 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:51:58.795077 10223 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:51:58.803273 10230 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:58.803306 10231 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:51:58.804333 10233 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:51:58.804760 10223 server_base.cc:1061] running on GCE node
I20260929 01:51:58.805545 10223 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:51:58.806931 10223 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:51:58.808135 10223 hybrid_clock.cc:648] HybridClock initialized: now 1790646718808097 us; error 35 us; skew 500 ppm
I20260929 01:51:58.808552 10223 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:51:58.810699 10223 webserver.cc:533] Webserver started at http://127.0.71.124:37331/ using document root <none> and password file <none>
I20260929 01:51:58.811404 10223 fs_manager.cc:362] Metadata directory not provided
I20260929 01:51:58.811518 10223 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:51:58.811869 10223 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:51:58.813915 10220 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } attempt: 4
W20260929 01:51:58.815896 10220 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:42795: Network error: Client connection negotiation failed: client connection to 127.0.71.124:42795: connect: Connection refused (error 111)
I20260929 01:51:58.816145 10223 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/data/instance:
uuid: "9ac72709b50c4e7588bebeae0e3cceff"
format_stamp: "Formatted at 2026-09-29 01:51:58 on dist-test-slave-vz23"
I20260929 01:51:58.816988 10223 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/wal/instance:
uuid: "9ac72709b50c4e7588bebeae0e3cceff"
format_stamp: "Formatted at 2026-09-29 01:51:58 on dist-test-slave-vz23"
I20260929 01:51:58.822355 10223 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:51:58.825819 10239 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:51:58.827337 10223 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:51:58.827533 10223 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "9ac72709b50c4e7588bebeae0e3cceff"
format_stamp: "Formatted at 2026-09-29 01:51:58 on dist-test-slave-vz23"
I20260929 01:51:58.827788 10223 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:51:58.844350 10223 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:51:58.845394 10223 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:51:58.845686 10223 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:51:58.867455 10223 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:42795
I20260929 01:51:58.867511 10290 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:42795 every 8 connection(s)
I20260929 01:51:58.869199 10223 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:51:58.873775 10291 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:51:58.878083 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 10223
I20260929 01:51:58.878342 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:51:58.879953 10291 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:51:58.892504 10149 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } attempt: 3
I20260929 01:51:58.895551 10291 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:51:58.904808 10291 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:51:58.906977 10149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: Bootstrap starting.
I20260929 01:51:58.911486 10149 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: Neither blocks nor log segments found. Creating new log.
I20260929 01:51:58.912581 10149 log.cc:826] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: Log is configured to *not* fsync() on all Append() calls
I20260929 01:51:58.914391 10291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff: Bootstrap starting.
I20260929 01:51:58.915972 10149 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: No bootstrap required, opened a new log
I20260929 01:51:58.917778 10291 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff: Neither blocks nor log segments found. Creating new log.
I20260929 01:51:58.918762 10291 log.cc:826] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff: Log is configured to *not* fsync() on all Append() calls
I20260929 01:51:58.921252 10291 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff: No bootstrap required, opened a new log
I20260929 01:51:58.921669 10149 raft_consensus.cc:359] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } }
I20260929 01:51:58.921921 10149 raft_consensus.cc:385] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:51:58.922051 10149 raft_consensus.cc:740] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee9c34aa5d8d476aaf1c0daadc8c59d6, State: Initialized, Role: FOLLOWER
I20260929 01:51:58.922672 10149 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } }
I20260929 01:51:58.923575 10300 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:51:58.923940 10300 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:58.925472 10149 sys_catalog.cc:565] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:51:58.927083 10291 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } }
I20260929 01:51:58.927376 10291 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:51:58.927522 10291 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ac72709b50c4e7588bebeae0e3cceff, State: Initialized, Role: FOLLOWER
I20260929 01:51:58.928104 10291 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } }
I20260929 01:51:58.929047 10305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:51:58.929319 10305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:58.930166 10291 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:51:58.938696 10313 catalog_manager.cc:1607] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:51:58.938903 10313 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 01:51:58.943392 10103 tablet.cc:2433] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:51:58.944095 10324 catalog_manager.cc:818] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 01:51:58.938211 10083 master_runner.cc:187] Error getting master registration for 127.0.71.125:46041: OK, instance_id { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" instance_seqno: 1790646718408580 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:51:58.948050 10083 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:51:58.949962 10223 master_runner.cc:187] Error getting master registration for 127.0.71.125:46041: OK, instance_id { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" instance_seqno: 1790646718408580 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:51:58.954231 10223 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:51:59.097867 10245 tablet.cc:2433] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:51:59.115156 10220 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } attempt: 5
I20260929 01:51:59.127398 10220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab: Bootstrap starting.
I20260929 01:51:59.132279 10220 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab: Neither blocks nor log segments found. Creating new log.
I20260929 01:51:59.134199 10220 log.cc:826] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab: Log is configured to *not* fsync() on all Append() calls
I20260929 01:51:59.137444 10220 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab: No bootstrap required, opened a new log
I20260929 01:51:59.145416 10220 raft_consensus.cc:359] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } }
I20260929 01:51:59.145774 10220 raft_consensus.cc:385] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:51:59.146003 10220 raft_consensus.cc:740] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd52acb9c5384217a5983864fb4140ab, State: Initialized, Role: FOLLOWER
I20260929 01:51:59.146762 10220 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } }
I20260929 01:51:59.147814 10326 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:51:59.148114 10326 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:59.149231 10220 sys_catalog.cc:565] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:51:59.161279 10337 catalog_manager.cc:1607] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:51:59.161507 10337 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:51:59.166646 10151 master_runner.cc:264] No leader master found from master fd52acb9c5384217a5983864fb4140ab
I20260929 01:51:59.166790 10151 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:51:59.318005 10305 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:51:59.318333 10305 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } }
I20260929 01:51:59.320199 10305 leader_election.cc:290] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ee9c34aa5d8d476aaf1c0daadc8c59d6 (127.0.71.126:44069), fd52acb9c5384217a5983864fb4140ab (127.0.71.125:46041)
I20260929 01:51:59.321287 10195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ac72709b50c4e7588bebeae0e3cceff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd52acb9c5384217a5983864fb4140ab" is_pre_election: true
I20260929 01:51:59.321292 10124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ac72709b50c4e7588bebeae0e3cceff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" is_pre_election: true
I20260929 01:51:59.321836 10195 raft_consensus.cc:2468] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ac72709b50c4e7588bebeae0e3cceff in term 0.
I20260929 01:51:59.321831 10124 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ac72709b50c4e7588bebeae0e3cceff in term 0.
I20260929 01:51:59.322577 10243 leader_election.cc:304] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ac72709b50c4e7588bebeae0e3cceff, fd52acb9c5384217a5983864fb4140ab; no voters:
I20260929 01:51:59.323287 10305 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:51:59.323416 10305 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:51:59.323506 10305 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:51:59.326972 10305 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } }
I20260929 01:51:59.327953 10124 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ac72709b50c4e7588bebeae0e3cceff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6"
I20260929 01:51:59.328145 10124 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:51:59.329383 10305 leader_election.cc:290] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [CANDIDATE]: Term 1 election: Requested vote from peers ee9c34aa5d8d476aaf1c0daadc8c59d6 (127.0.71.126:44069), fd52acb9c5384217a5983864fb4140ab (127.0.71.125:46041)
I20260929 01:51:59.330089 10195 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ac72709b50c4e7588bebeae0e3cceff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd52acb9c5384217a5983864fb4140ab"
I20260929 01:51:59.330318 10195 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:51:59.331861 10124 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ac72709b50c4e7588bebeae0e3cceff in term 1.
I20260929 01:51:59.332472 10242 leader_election.cc:304] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ac72709b50c4e7588bebeae0e3cceff, ee9c34aa5d8d476aaf1c0daadc8c59d6; no voters:
I20260929 01:51:59.332918 10305 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:51:59.334578 10195 raft_consensus.cc:2468] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ac72709b50c4e7588bebeae0e3cceff in term 1.
I20260929 01:51:59.335603 10305 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [term 1 LEADER]: Becoming Leader. State: Replica: 9ac72709b50c4e7588bebeae0e3cceff, State: Running, Role: LEADER
I20260929 01:51:59.336201 10305 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } }
I20260929 01:51:59.341281 10339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ac72709b50c4e7588bebeae0e3cceff. Latest consensus state: current_term: 1 leader_uuid: "9ac72709b50c4e7588bebeae0e3cceff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:51:59.341650 10339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [sys.catalog]: This master's current role is: LEADER
I20260929 01:51:59.343171 10341 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:51:59.348996 10341 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:51:59.367313 10124 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [term 1 FOLLOWER]: Refusing update from remote peer 9ac72709b50c4e7588bebeae0e3cceff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:51:59.367462 10195 raft_consensus.cc:1275] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [term 1 FOLLOWER]: Refusing update from remote peer 9ac72709b50c4e7588bebeae0e3cceff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:51:59.368383 10339 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:51:59.368798 10305 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 01:51:59.395323 10174 tablet.cc:2433] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:51:59.397301 10300 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ac72709b50c4e7588bebeae0e3cceff. Latest consensus state: current_term: 1 leader_uuid: "9ac72709b50c4e7588bebeae0e3cceff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:51:59.397603 10300 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:59.401036 10343 mvcc.cc:204] Tried to move back new op lower bound from 7334488962518147072 to 7334488962406174720. Current Snapshot: MvccSnapshot[applied={T|T < 7334488962518147072}]
I20260929 01:51:59.409067 10300 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ac72709b50c4e7588bebeae0e3cceff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:51:59.409400 10300 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:59.410676 10344 mvcc.cc:204] Tried to move back new op lower bound from 7334488962518147072 to 7334488962406174720. Current Snapshot: MvccSnapshot[applied={T|T < 7334488962518147072 or (T in {7334488962518147072})}]
I20260929 01:51:59.413691 10326 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ac72709b50c4e7588bebeae0e3cceff. Latest consensus state: current_term: 1 leader_uuid: "9ac72709b50c4e7588bebeae0e3cceff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:51:59.413978 10326 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:59.414211 10305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ac72709b50c4e7588bebeae0e3cceff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:51:59.414415 10305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [sys.catalog]: This master's current role is: LEADER
I20260929 01:51:59.416870 10341 catalog_manager.cc:1396] Generated new cluster ID: 63f4e7e6fa36411ea78e33545a86e63f
I20260929 01:51:59.416973 10341 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:51:59.417464 10339 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ac72709b50c4e7588bebeae0e3cceff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:51:59.417743 10339 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [sys.catalog]: This master's current role is: LEADER
I20260929 01:51:59.426247 10345 mvcc.cc:204] Tried to move back new op lower bound from 7334488962518147072 to 7334488962406174720. Current Snapshot: MvccSnapshot[applied={T|T < 7334488962518147072}]
I20260929 01:51:59.444216 10326 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9ac72709b50c4e7588bebeae0e3cceff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:51:59.444470 10326 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:51:59.483047 10341 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 01:51:59.485466 10341 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:51:59.513767 10341 catalog_manager.cc:6219] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff: Generated new TSK 0
I20260929 01:51:59.515064 10341 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:51:59.556175 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--builtin_ntp_servers=127.0.71.84:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:51:59.941442 10313 catalog_manager.cc:1308] Loaded cluster ID: 63f4e7e6fa36411ea78e33545a86e63f
I20260929 01:51:59.941570 10313 catalog_manager.cc:1601] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: loading cluster ID for follower catalog manager: success
I20260929 01:51:59.945624 10313 catalog_manager.cc:1623] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: acquiring CA information for follower catalog manager: success
I20260929 01:51:59.948261 10313 catalog_manager.cc:1651] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 01:52:00.003422 10352 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:00.003898 10352 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:00.004040 10352 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:00.013314 10352 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:00.013537 10352 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:52:00.023639 10352 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:00.025374 10352 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:00.027276 10352 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:00.037217 10358 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:00.037400 10360 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:00.037427 10357 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:00.038594 10352 server_base.cc:1061] running on GCE node
I20260929 01:52:00.039323 10352 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:00.040295 10352 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:00.041499 10352 hybrid_clock.cc:648] HybridClock initialized: now 1790646720041450 us; error 64 us; skew 500 ppm
I20260929 01:52:00.041836 10352 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:00.043964 10352 webserver.cc:533] Webserver started at http://127.0.71.65:34629/ using document root <none> and password file <none>
I20260929 01:52:00.044785 10352 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:00.044904 10352 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:00.045287 10352 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:52:00.047766 10352 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "e2615c14e9644ce1a653d800426a3c94"
format_stamp: "Formatted at 2026-09-29 01:52:00 on dist-test-slave-vz23"
I20260929 01:52:00.048595 10352 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "e2615c14e9644ce1a653d800426a3c94"
format_stamp: "Formatted at 2026-09-29 01:52:00 on dist-test-slave-vz23"
I20260929 01:52:00.054631 10352 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 01:52:00.058184 10366 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:00.059532 10352 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:52:00.059728 10352 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "e2615c14e9644ce1a653d800426a3c94"
format_stamp: "Formatted at 2026-09-29 01:52:00 on dist-test-slave-vz23"
I20260929 01:52:00.059950 10352 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:00.089131 10352 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:00.090071 10352 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:00.090360 10352 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:00.091481 10352 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:52:00.093531 10352 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:52:00.093617 10352 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:00.093732 10352 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:52:00.093808 10352 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:00.130069 10352 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:46613
I20260929 01:52:00.130134 10478 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:46613 every 8 connection(s)
I20260929 01:52:00.131618 10352 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:52:00.133702 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 10352
I20260929 01:52:00.133880 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
I20260929 01:52:00.139696 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.66:0
--local_ip_for_outbound_sockets=127.0.71.66
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--builtin_ntp_servers=127.0.71.84:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:52:00.164117 10337 catalog_manager.cc:1308] Loaded cluster ID: 63f4e7e6fa36411ea78e33545a86e63f
I20260929 01:52:00.164247 10337 catalog_manager.cc:1601] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab: loading cluster ID for follower catalog manager: success
I20260929 01:52:00.168849 10337 catalog_manager.cc:1623] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab: acquiring CA information for follower catalog manager: success
I20260929 01:52:00.170229 10479 heartbeater.cc:344] Connected to a master server at 127.0.71.124:42795
I20260929 01:52:00.170347 10481 heartbeater.cc:344] Connected to a master server at 127.0.71.125:46041
I20260929 01:52:00.170653 10479 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:00.170670 10481 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:00.171912 10481 heartbeater.cc:507] Master 127.0.71.125:46041 requested a full tablet report, sending...
I20260929 01:52:00.171888 10479 heartbeater.cc:507] Master 127.0.71.124:42795 requested a full tablet report, sending...
I20260929 01:52:00.173070 10337 catalog_manager.cc:1651] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:52:00.174206 10256 ts_manager.cc:194] Registered new tserver with Master: e2615c14e9644ce1a653d800426a3c94 (127.0.71.65:46613)
I20260929 01:52:00.174607 10185 ts_manager.cc:194] Registered new tserver with Master: e2615c14e9644ce1a653d800426a3c94 (127.0.71.65:46613)
I20260929 01:52:00.175061 10480 heartbeater.cc:344] Connected to a master server at 127.0.71.126:44069
I20260929 01:52:00.175185 10480 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:00.175716 10480 heartbeater.cc:507] Master 127.0.71.126:44069 requested a full tablet report, sending...
I20260929 01:52:00.176036 10256 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:60303
I20260929 01:52:00.177471 10114 ts_manager.cc:194] Registered new tserver with Master: e2615c14e9644ce1a653d800426a3c94 (127.0.71.65:46613)
W20260929 01:52:00.491532 10488 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:00.491856 10488 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:00.491930 10488 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:00.500576 10488 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:00.500743 10488 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.66
I20260929 01:52:00.510614 10488 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:00.512111 10488 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:00.513830 10488 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:00.523582 10494 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:00.523586 10495 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:00.524219 10497 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:00.525050 10488 server_base.cc:1061] running on GCE node
I20260929 01:52:00.525687 10488 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:00.526832 10488 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:00.528088 10488 hybrid_clock.cc:648] HybridClock initialized: now 1790646720528032 us; error 46 us; skew 500 ppm
I20260929 01:52:00.528411 10488 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:00.530860 10488 webserver.cc:533] Webserver started at http://127.0.71.66:40109/ using document root <none> and password file <none>
I20260929 01:52:00.531638 10488 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:00.531780 10488 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:00.532155 10488 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:52:00.534580 10488 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/data/instance:
uuid: "1e05a693a585490cb0fe904ba5e7c35e"
format_stamp: "Formatted at 2026-09-29 01:52:00 on dist-test-slave-vz23"
I20260929 01:52:00.535424 10488 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/wal/instance:
uuid: "1e05a693a585490cb0fe904ba5e7c35e"
format_stamp: "Formatted at 2026-09-29 01:52:00 on dist-test-slave-vz23"
I20260929 01:52:00.540652 10488 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 01:52:00.544003 10503 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:00.545701 10488 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:52:00.545874 10488 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/wal
uuid: "1e05a693a585490cb0fe904ba5e7c35e"
format_stamp: "Formatted at 2026-09-29 01:52:00 on dist-test-slave-vz23"
I20260929 01:52:00.546038 10488 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:00.568056 10488 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:00.569031 10488 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:00.569324 10488 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:00.570286 10488 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:52:00.571961 10488 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:52:00.572043 10488 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:00.572119 10488 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:52:00.572166 10488 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:00.607133 10488 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.66:35959
I20260929 01:52:00.607219 10615 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.66:35959 every 8 connection(s)
I20260929 01:52:00.608924 10488 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
I20260929 01:52:00.611451 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 10488
I20260929 01:52:00.611696 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/wal/instance
I20260929 01:52:00.621270 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.67:0
--local_ip_for_outbound_sockets=127.0.71.67
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--builtin_ntp_servers=127.0.71.84:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:52:00.628325 10616 heartbeater.cc:344] Connected to a master server at 127.0.71.124:42795
I20260929 01:52:00.628834 10616 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:00.629997 10616 heartbeater.cc:507] Master 127.0.71.124:42795 requested a full tablet report, sending...
I20260929 01:52:00.632380 10256 ts_manager.cc:194] Registered new tserver with Master: 1e05a693a585490cb0fe904ba5e7c35e (127.0.71.66:35959)
I20260929 01:52:00.633451 10256 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.66:33547
I20260929 01:52:00.634382 10618 heartbeater.cc:344] Connected to a master server at 127.0.71.125:46041
I20260929 01:52:00.634544 10618 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:00.635213 10618 heartbeater.cc:507] Master 127.0.71.125:46041 requested a full tablet report, sending...
I20260929 01:52:00.636314 10185 ts_manager.cc:194] Registered new tserver with Master: 1e05a693a585490cb0fe904ba5e7c35e (127.0.71.66:35959)
I20260929 01:52:00.637784 10617 heartbeater.cc:344] Connected to a master server at 127.0.71.126:44069
I20260929 01:52:00.637923 10617 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:00.638242 10617 heartbeater.cc:507] Master 127.0.71.126:44069 requested a full tablet report, sending...
I20260929 01:52:00.639264 10114 ts_manager.cc:194] Registered new tserver with Master: 1e05a693a585490cb0fe904ba5e7c35e (127.0.71.66:35959)
W20260929 01:52:00.959951 10623 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:00.960242 10623 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:00.960314 10623 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:00.973733 10623 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:00.973956 10623 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.67
I20260929 01:52:00.984961 10623 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:00.986474 10623 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:00.988080 10623 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:00.997771 10628 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:00.997959 10629 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:00.998035 10631 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:00.998627 10623 server_base.cc:1061] running on GCE node
I20260929 01:52:00.999261 10623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:01.000304 10623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:01.001520 10623 hybrid_clock.cc:648] HybridClock initialized: now 1790646721001406 us; error 109 us; skew 500 ppm
I20260929 01:52:01.001899 10623 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:01.006219 10623 webserver.cc:533] Webserver started at http://127.0.71.67:39747/ using document root <none> and password file <none>
I20260929 01:52:01.007282 10623 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:01.007401 10623 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:01.007848 10623 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:52:01.010855 10623 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/data/instance:
uuid: "b76557c513e346578fbdae008c861b48"
format_stamp: "Formatted at 2026-09-29 01:52:01 on dist-test-slave-vz23"
I20260929 01:52:01.011662 10623 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/wal/instance:
uuid: "b76557c513e346578fbdae008c861b48"
format_stamp: "Formatted at 2026-09-29 01:52:01 on dist-test-slave-vz23"
I20260929 01:52:01.017228 10623 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 01:52:01.020768 10637 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:01.022181 10623 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:52:01.022405 10623 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/wal
uuid: "b76557c513e346578fbdae008c861b48"
format_stamp: "Formatted at 2026-09-29 01:52:01 on dist-test-slave-vz23"
I20260929 01:52:01.022629 10623 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:01.050243 10623 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:01.051215 10623 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:01.051505 10623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:01.052493 10623 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:52:01.054229 10623 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:52:01.054311 10623 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:01.054459 10623 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:52:01.054549 10623 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:01.089480 10623 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.67:35149
I20260929 01:52:01.089569 10749 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.67:35149 every 8 connection(s)
I20260929 01:52:01.090967 10623 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
I20260929 01:52:01.101330 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 10623
I20260929 01:52:01.101589 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/wal/instance
I20260929 01:52:01.107741 10751 heartbeater.cc:344] Connected to a master server at 127.0.71.126:44069
I20260929 01:52:01.108114 10751 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:01.109329 10751 heartbeater.cc:507] Master 127.0.71.126:44069 requested a full tablet report, sending...
I20260929 01:52:01.109768 10752 heartbeater.cc:344] Connected to a master server at 127.0.71.125:46041
I20260929 01:52:01.109915 10752 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:01.110421 10752 heartbeater.cc:507] Master 127.0.71.125:46041 requested a full tablet report, sending...
I20260929 01:52:01.110513 10750 heartbeater.cc:344] Connected to a master server at 127.0.71.124:42795
I20260929 01:52:01.110648 10750 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:01.111094 10750 heartbeater.cc:507] Master 127.0.71.124:42795 requested a full tablet report, sending...
I20260929 01:52:01.111222 10114 ts_manager.cc:194] Registered new tserver with Master: b76557c513e346578fbdae008c861b48 (127.0.71.67:35149)
I20260929 01:52:01.112556 10185 ts_manager.cc:194] Registered new tserver with Master: b76557c513e346578fbdae008c861b48 (127.0.71.67:35149)
I20260929 01:52:01.112557 10256 ts_manager.cc:194] Registered new tserver with Master: b76557c513e346578fbdae008c861b48 (127.0.71.67:35149)
I20260929 01:52:01.113404 10256 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.67:44303
I20260929 01:52:01.129115 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:52:01.136999 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:52:01.137238 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 10083
I20260929 01:52:01.138055 10144 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:01.178571 10479 heartbeater.cc:499] Master 127.0.71.124:42795 was elected leader, sending a full tablet report...
I20260929 01:52:01.246652 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 10083
I20260929 01:52:01.267190 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:44069
--webserver_interface=127.0.71.126
--webserver_port=36963
--builtin_ntp_servers=127.0.71.84:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795 with env {}
W20260929 01:52:01.415228 10242 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.126:44069: connect: Connection refused (error 111)
W20260929 01:52:01.417838 10242 consensus_peers.cc:597] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff -> Peer ee9c34aa5d8d476aaf1c0daadc8c59d6 (127.0.71.126:44069): Couldn't send request to peer ee9c34aa5d8d476aaf1c0daadc8c59d6. Status: Network error: Client connection negotiation failed: client connection to 127.0.71.126:44069: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260929 01:52:01.600574 10758 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:01.600950 10758 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:01.601084 10758 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:01.610138 10758 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:52:01.610257 10758 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:01.610371 10758 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:52:01.610452 10758 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:52:01.620748 10758 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:44069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=36963
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:01.622388 10758 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:01.624217 10758 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:01.632712 10765 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:01.632823 10766 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:01.632905 10768 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:01.634141 10758 server_base.cc:1061] running on GCE node
I20260929 01:52:01.635160 10758 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:01.636811 10758 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:01.637238 10616 heartbeater.cc:499] Master 127.0.71.124:42795 was elected leader, sending a full tablet report...
I20260929 01:52:01.638386 10758 hybrid_clock.cc:648] HybridClock initialized: now 1790646721638155 us; error 218 us; skew 500 ppm
I20260929 01:52:01.638810 10758 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20260929 01:52:01.641395 10617 heartbeater.cc:646] Failed to heartbeat to 127.0.71.126:44069 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.71.126:44069: connect: Connection refused (error 111)
I20260929 01:52:01.641463 10758 webserver.cc:533] Webserver started at http://127.0.71.126:36963/ using document root <none> and password file <none>
I20260929 01:52:01.642162 10758 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:01.642261 10758 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:01.647665 10758 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:52:01.650949 10775 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:01.652899 10758 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:52:01.653072 10758 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6"
format_stamp: "Formatted at 2026-09-29 01:51:57 on dist-test-slave-vz23"
I20260929 01:52:01.653780 10758 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:01.670002 10758 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:01.670792 10758 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:01.671103 10758 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:01.692076 10758 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:44069
I20260929 01:52:01.692116 10826 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:44069 every 8 connection(s)
I20260929 01:52:01.693851 10758 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:52:01.696635 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 10758
I20260929 01:52:01.696955 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 10352
I20260929 01:52:01.701771 10827 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:52:01.705961 10827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: Bootstrap starting.
I20260929 01:52:01.709060 10474 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:01.729324 10827 log.cc:826] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: Log is configured to *not* fsync() on all Append() calls
I20260929 01:52:01.739984 10827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:52:01.740636 10827 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: Bootstrap complete.
I20260929 01:52:01.750810 10827 raft_consensus.cc:359] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } }
I20260929 01:52:01.751765 10827 raft_consensus.cc:740] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee9c34aa5d8d476aaf1c0daadc8c59d6, State: Initialized, Role: FOLLOWER
I20260929 01:52:01.752655 10827 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } }
I20260929 01:52:01.753547 10833 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:52:01.753827 10833 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:01.754859 10827 sys_catalog.cc:565] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:52:01.765321 10844 catalog_manager.cc:1308] Loaded cluster ID: 63f4e7e6fa36411ea78e33545a86e63f
I20260929 01:52:01.765431 10844 catalog_manager.cc:1601] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: loading cluster ID for follower catalog manager: success
I20260929 01:52:01.769836 10844 catalog_manager.cc:1623] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: acquiring CA information for follower catalog manager: success
I20260929 01:52:01.772565 10844 catalog_manager.cc:1651] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:52:01.829563 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 10352
I20260929 01:52:01.852057 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:46613
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--webserver_port=34629
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:52:02.025787 10305 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ac72709b50c4e7588bebeae0e3cceff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:52:02.026539 10305 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [sys.catalog]: This master's current role is: LEADER
I20260929 01:52:02.036468 10833 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ac72709b50c4e7588bebeae0e3cceff. Latest consensus state: current_term: 1 leader_uuid: "9ac72709b50c4e7588bebeae0e3cceff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:52:02.037211 10833 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:02.115284 10750 heartbeater.cc:499] Master 127.0.71.124:42795 was elected leader, sending a full tablet report...
I20260929 01:52:02.125950 10792 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "b76557c513e346578fbdae008c861b48" instance_seqno: 1790646721081076) as {username='slave'} at 127.0.71.67:37417; Asking this server to re-register.
I20260929 01:52:02.126914 10751 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:02.128912 10792 ts_manager.cc:194] Registered new tserver with Master: b76557c513e346578fbdae008c861b48 (127.0.71.67:35149)
W20260929 01:52:02.199527 10850 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:02.199853 10850 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:02.200021 10850 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:02.208528 10850 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:02.208758 10850 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:52:02.218892 10850 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:46613
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=34629
--tserver_master_addrs=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:02.220552 10850 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:02.222337 10850 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:02.232291 10861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:02.232789 10860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:02.233579 10863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:02.233822 10850 server_base.cc:1061] running on GCE node
I20260929 01:52:02.234472 10850 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:02.235525 10850 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:02.236809 10850 hybrid_clock.cc:648] HybridClock initialized: now 1790646722236803 us; error 82 us; skew 500 ppm
I20260929 01:52:02.237139 10850 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:02.239434 10850 webserver.cc:533] Webserver started at http://127.0.71.65:34629/ using document root <none> and password file <none>
I20260929 01:52:02.240190 10850 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:02.240339 10850 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:02.245885 10850 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:52:02.248661 10869 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:02.250026 10850 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:52:02.250209 10850 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "e2615c14e9644ce1a653d800426a3c94"
format_stamp: "Formatted at 2026-09-29 01:52:00 on dist-test-slave-vz23"
I20260929 01:52:02.250983 10850 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:02.267341 10850 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:02.268198 10850 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:02.268481 10850 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:02.269382 10850 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:52:02.271090 10850 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:52:02.271191 10850 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:02.271392 10850 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:52:02.271469 10850 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:02.307525 10850 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:46613
I20260929 01:52:02.307562 10981 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:46613 every 8 connection(s)
I20260929 01:52:02.309110 10850 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:52:02.312263 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 10850
I20260929 01:52:02.330825 10984 heartbeater.cc:344] Connected to a master server at 127.0.71.125:46041
I20260929 01:52:02.330948 10983 heartbeater.cc:344] Connected to a master server at 127.0.71.126:44069
I20260929 01:52:02.331323 10983 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:02.331331 10984 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:02.332391 10983 heartbeater.cc:507] Master 127.0.71.126:44069 requested a full tablet report, sending...
I20260929 01:52:02.332993 10984 heartbeater.cc:507] Master 127.0.71.125:46041 requested a full tablet report, sending...
I20260929 01:52:02.334498 10792 ts_manager.cc:194] Registered new tserver with Master: e2615c14e9644ce1a653d800426a3c94 (127.0.71.65:46613)
I20260929 01:52:02.334620 10982 heartbeater.cc:344] Connected to a master server at 127.0.71.124:42795
I20260929 01:52:02.334739 10982 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:02.335212 10982 heartbeater.cc:507] Master 127.0.71.124:42795 requested a full tablet report, sending...
I20260929 01:52:02.335702 10185 ts_manager.cc:194] Re-registered known tserver with Master: e2615c14e9644ce1a653d800426a3c94 (127.0.71.65:46613)
I20260929 01:52:02.336280 10256 ts_manager.cc:194] Re-registered known tserver with Master: e2615c14e9644ce1a653d800426a3c94 (127.0.71.65:46613)
I20260929 01:52:02.337342 10256 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:45261
I20260929 01:52:02.355964 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 10850
I20260929 01:52:02.368816 10977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:02.456157 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 10850
I20260929 01:52:02.472962 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:46613
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--webserver_port=34629
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {}
I20260929 01:52:02.548552 10851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9ac72709b50c4e7588bebeae0e3cceff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:52:02.548846 10851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [sys.catalog]: This master's current role is: LEADER
I20260929 01:52:02.652045 10617 heartbeater.cc:344] Connected to a master server at 127.0.71.126:44069
I20260929 01:52:02.653266 10792 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1e05a693a585490cb0fe904ba5e7c35e" instance_seqno: 1790646720599293) as {username='slave'} at 127.0.71.66:44631; Asking this server to re-register.
I20260929 01:52:02.653820 10617 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:02.654925 10792 ts_manager.cc:194] Registered new tserver with Master: 1e05a693a585490cb0fe904ba5e7c35e (127.0.71.66:35959)
W20260929 01:52:02.807924 10995 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:02.808284 10995 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:02.808470 10995 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:52:02.808565 10995 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:02.817756 10995 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:02.817996 10995 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:52:02.829277 10995 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:46613
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=34629
--tserver_master_addrs=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:02.831058 10995 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:02.833081 10995 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:02.843668 11005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:02.843715 11004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:02.844148 11007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:02.845194 10995 server_base.cc:1061] running on GCE node
I20260929 01:52:02.845763 10995 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:02.846696 10995 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:02.847991 10995 hybrid_clock.cc:648] HybridClock initialized: now 1790646722847926 us; error 53 us; skew 500 ppm
I20260929 01:52:02.848327 10995 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:02.850737 10995 webserver.cc:533] Webserver started at http://127.0.71.65:34629/ using document root <none> and password file <none>
I20260929 01:52:02.851569 10995 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:02.851688 10995 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:02.857482 10995 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260929 01:52:02.860343 11013 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:02.862061 10995 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:52:02.862293 10995 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "e2615c14e9644ce1a653d800426a3c94"
format_stamp: "Formatted at 2026-09-29 01:52:00 on dist-test-slave-vz23"
I20260929 01:52:02.863067 10995 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:02.884600 10995 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:02.885484 10995 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:02.885784 10995 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:02.886729 10995 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:52:02.888695 10995 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:52:02.888839 10995 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:02.888995 10995 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:52:02.889101 10995 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:52:02.889565 10995 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:52:02.889693 10995 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:52:02.889787 10995 fault_injection.cc:59] injecting fault for kudu (pid 10995): FLAGS_fault_before_start (process will exit)
I20260929 01:52:02.912827 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:46613
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--webserver_port=34629
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:38781
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260929 01:52:03.254053 11021 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:03.254434 11021 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:03.254588 11021 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:03.263376 11021 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:03.263604 11021 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:52:03.273815 11021 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38781
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:46613
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=34629
--tserver_master_addrs=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:03.275472 11021 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:03.277281 11021 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:03.287460 11026 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:03.287484 11027 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:03.288175 11029 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:03.288722 11021 server_base.cc:1061] running on GCE node
I20260929 01:52:03.289350 11021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:03.290519 11021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:03.291770 11021 hybrid_clock.cc:648] HybridClock initialized: now 1790646723291700 us; error 62 us; skew 500 ppm
I20260929 01:52:03.292105 11021 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:03.294440 11021 webserver.cc:533] Webserver started at http://127.0.71.65:34629/ using document root <none> and password file <none>
I20260929 01:52:03.295236 11021 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:03.295377 11021 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:03.301071 11021 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 01:52:03.303915 11035 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:03.305470 11021 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:52:03.305691 11021 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "e2615c14e9644ce1a653d800426a3c94"
format_stamp: "Formatted at 2026-09-29 01:52:00 on dist-test-slave-vz23"
I20260929 01:52:03.306417 11021 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:03.332790 11021 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:03.333730 11021 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:03.333983 11021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:03.335045 11021 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:52:03.336791 11021 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:52:03.336865 11021 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:03.336947 11021 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:52:03.336990 11021 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:03.372651 11021 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:46613
I20260929 01:52:03.372726 11147 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:46613 every 8 connection(s)
I20260929 01:52:03.374150 11021 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:52:03.382827 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 11021
I20260929 01:52:03.383168 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 11021
I20260929 01:52:03.391996 11148 heartbeater.cc:344] Connected to a master server at 127.0.71.124:42795
I20260929 01:52:03.392416 11148 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:03.393579 11148 heartbeater.cc:507] Master 127.0.71.124:42795 requested a full tablet report, sending...
I20260929 01:52:03.393719 11149 heartbeater.cc:344] Connected to a master server at 127.0.71.126:44069
I20260929 01:52:03.393893 11149 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:03.394032 11150 heartbeater.cc:344] Connected to a master server at 127.0.71.125:46041
I20260929 01:52:03.394186 11150 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:03.394523 11149 heartbeater.cc:507] Master 127.0.71.126:44069 requested a full tablet report, sending...
I20260929 01:52:03.394687 11150 heartbeater.cc:507] Master 127.0.71.125:46041 requested a full tablet report, sending...
I20260929 01:52:03.395666 10185 ts_manager.cc:194] Re-registered known tserver with Master: e2615c14e9644ce1a653d800426a3c94 (127.0.71.65:46613)
I20260929 01:52:03.395929 10256 ts_manager.cc:194] Re-registered known tserver with Master: e2615c14e9644ce1a653d800426a3c94 (127.0.71.65:46613)
I20260929 01:52:03.396513 10792 ts_manager.cc:194] Re-registered known tserver with Master: e2615c14e9644ce1a653d800426a3c94 (127.0.71.65:46613)
I20260929 01:52:03.397141 10256 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:41703
I20260929 01:52:03.398999 11143 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:03.492816 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 11021
I20260929 01:52:03.509259 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 10488
I20260929 01:52:03.517509 10611 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:03.604923 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 10488
I20260929 01:52:03.621726 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 10623
I20260929 01:52:03.629947 10745 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:03.718581 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 10623
I20260929 01:52:03.735293 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 10758
I20260929 01:52:03.742262 10822 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:03.849298 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 10758
I20260929 01:52:03.868710 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 10151
I20260929 01:52:03.870113 10215 generic_service.cc:196] Checking for leaks (request via RPC)
W20260929 01:52:03.933362 10242 consensus_peers.cc:597] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff -> Peer ee9c34aa5d8d476aaf1c0daadc8c59d6 (127.0.71.126:44069): Couldn't send request to peer ee9c34aa5d8d476aaf1c0daadc8c59d6. Status: Network error: Client connection negotiation failed: client connection to 127.0.71.126:44069: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 01:52:03.955595 10223 master_runner.cc:187] Error getting master registration for 127.0.71.126:44069: Network error: Client connection negotiation failed: client connection to 127.0.71.126:44069: connect: Connection refused (error 111),
I20260929 01:52:03.955843 10223 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:52:03.979431 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 10151
I20260929 01:52:03.999418 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 10223
I20260929 01:52:04.000857 10286 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:04.113232 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 10223
Shutting down hive metastore.
2026-09-29T01:52:04Z chronyd exiting
2026-09-29T01:52:04Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:52:04Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/security.properties
2026-09-29 01:52:05: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260929 01:52:13.670993 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:44069
--webserver_interface=127.0.71.126
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:45381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795 with env {}
W20260929 01:52:14.000355 11351 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:14.000775 11351 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:14.000852 11351 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:14.009665 11351 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:52:14.009760 11351 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:14.009816 11351 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:52:14.009856 11351 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:52:14.019941 11351 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:45381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:44069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:14.021582 11351 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:14.023301 11351 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:14.031419 11357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:14.031452 11356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:14.031936 11359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:14.032517 11351 server_base.cc:1061] running on GCE node
I20260929 01:52:14.033250 11351 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:14.034662 11351 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:14.035888 11351 hybrid_clock.cc:648] HybridClock initialized: now 1790646734035832 us; error 51 us; skew 500 ppm
I20260929 01:52:14.036247 11351 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:14.038532 11351 webserver.cc:533] Webserver started at http://127.0.71.126:46033/ using document root <none> and password file <none>
I20260929 01:52:14.039254 11351 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:14.039393 11351 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:14.044930 11351 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260929 01:52:14.047817 11365 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:14.049194 11351 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:52:14.049381 11351 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6"
format_stamp: "Formatted at 2026-09-29 01:51:57 on dist-test-slave-vz23"
I20260929 01:52:14.050127 11351 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:14.108341 11351 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:14.109314 11351 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:14.109623 11351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:14.130683 11416 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:44069 every 8 connection(s)
I20260929 01:52:14.130689 11351 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:44069
I20260929 01:52:14.132380 11351 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:52:14.139168 11417 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:52:14.140928 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 11351
I20260929 01:52:14.141119 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:52:14.142043 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:46041
--webserver_interface=127.0.71.125
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:45381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795 with env {}
I20260929 01:52:14.142661 11417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: Bootstrap starting.
I20260929 01:52:14.164249 11417 log.cc:826] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: Log is configured to *not* fsync() on all Append() calls
I20260929 01:52:14.175308 11417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:52:14.175944 11417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: Bootstrap complete.
I20260929 01:52:14.185280 11417 raft_consensus.cc:359] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } }
I20260929 01:52:14.186229 11417 raft_consensus.cc:740] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee9c34aa5d8d476aaf1c0daadc8c59d6, State: Initialized, Role: FOLLOWER
I20260929 01:52:14.187044 11417 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } }
I20260929 01:52:14.187884 11421 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:52:14.188195 11421 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:14.189155 11417 sys_catalog.cc:565] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:52:14.200487 11432 catalog_manager.cc:1308] Loaded cluster ID: 63f4e7e6fa36411ea78e33545a86e63f
I20260929 01:52:14.200735 11432 catalog_manager.cc:1601] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: loading cluster ID for follower catalog manager: success
I20260929 01:52:14.205962 11432 catalog_manager.cc:1623] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: acquiring CA information for follower catalog manager: success
I20260929 01:52:14.208751 11432 catalog_manager.cc:1651] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:52:14.223459 11351 master_runner.cc:187] Error getting master registration for 127.0.71.125:46041: Network error: Client connection negotiation failed: client connection to 127.0.71.125:46041: connect: Connection refused (error 111),
I20260929 01:52:14.229557 11351 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:52:14.507196 11419 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:14.507743 11419 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:14.507929 11419 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:14.522120 11419 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:52:14.522246 11419 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:14.522374 11419 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:52:14.522459 11419 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:52:14.537810 11419 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:45381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:46041
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:14.540023 11419 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:14.541877 11419 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:14.550385 11440 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:14.552249 11441 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:14.556742 11419 server_base.cc:1061] running on GCE node
W20260929 01:52:14.557439 11443 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:14.558535 11419 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:14.560400 11419 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:14.561685 11419 hybrid_clock.cc:648] HybridClock initialized: now 1790646734561617 us; error 63 us; skew 500 ppm
I20260929 01:52:14.562079 11419 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:14.564711 11419 webserver.cc:533] Webserver started at http://127.0.71.125:34473/ using document root <none> and password file <none>
I20260929 01:52:14.565622 11419 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:14.565745 11419 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:14.572906 11419 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 01:52:14.576318 11449 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:14.578078 11419 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:52:14.578262 11419 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "fd52acb9c5384217a5983864fb4140ab"
format_stamp: "Formatted at 2026-09-29 01:51:58 on dist-test-slave-vz23"
I20260929 01:52:14.579201 11419 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:14.625041 11419 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:14.625901 11419 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:14.626163 11419 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:14.647017 11419 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:46041
I20260929 01:52:14.647043 11500 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:46041 every 8 connection(s)
I20260929 01:52:14.648662 11419 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:52:14.655692 11501 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:52:14.655710 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 11419
I20260929 01:52:14.655905 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:52:14.656894 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:42795
--webserver_interface=127.0.71.124
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:45381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795 with env {}
I20260929 01:52:14.659029 11501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab: Bootstrap starting.
I20260929 01:52:14.675005 11501 log.cc:826] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab: Log is configured to *not* fsync() on all Append() calls
I20260929 01:52:14.683295 11501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:52:14.683821 11501 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab: Bootstrap complete.
I20260929 01:52:14.690954 11501 raft_consensus.cc:359] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } }
I20260929 01:52:14.691952 11501 raft_consensus.cc:740] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd52acb9c5384217a5983864fb4140ab, State: Initialized, Role: FOLLOWER
I20260929 01:52:14.692785 11501 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } }
I20260929 01:52:14.693708 11505 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:52:14.694008 11505 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:14.694875 11501 sys_catalog.cc:565] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:52:14.705272 11516 catalog_manager.cc:1308] Loaded cluster ID: 63f4e7e6fa36411ea78e33545a86e63f
I20260929 01:52:14.705489 11516 catalog_manager.cc:1601] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab: loading cluster ID for follower catalog manager: success
I20260929 01:52:14.710944 11516 catalog_manager.cc:1623] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab: acquiring CA information for follower catalog manager: success
I20260929 01:52:14.714171 11516 catalog_manager.cc:1651] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:52:14.735466 11419 master_runner.cc:187] Error getting master registration for 127.0.71.124:42795: Network error: Client connection negotiation failed: client connection to 127.0.71.124:42795: connect: Connection refused (error 111),
I20260929 01:52:14.739336 11419 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:52:15.023044 11503 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:15.023362 11503 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:15.023516 11503 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:15.032315 11503 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:52:15.032405 11503 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:15.032522 11503 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:52:15.032591 11503 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:52:15.043422 11503 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:45381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:42795
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:15.045195 11503 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:15.047142 11503 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:15.055892 11525 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:15.055852 11524 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:15.057418 11527 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:15.058768 11503 server_base.cc:1061] running on GCE node
I20260929 01:52:15.059666 11503 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:15.061427 11503 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:15.062765 11503 hybrid_clock.cc:648] HybridClock initialized: now 1790646735062711 us; error 44 us; skew 500 ppm
I20260929 01:52:15.063195 11503 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:15.065812 11503 webserver.cc:533] Webserver started at http://127.0.71.124:34595/ using document root <none> and password file <none>
I20260929 01:52:15.066599 11503 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:15.066740 11503 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:15.073088 11503 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:52:15.077122 11533 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:15.078893 11503 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:52:15.079103 11503 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "9ac72709b50c4e7588bebeae0e3cceff"
format_stamp: "Formatted at 2026-09-29 01:51:58 on dist-test-slave-vz23"
I20260929 01:52:15.080029 11503 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:15.100570 11503 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:15.101729 11503 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:15.102079 11503 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:15.125741 11503 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:42795
I20260929 01:52:15.125744 11584 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:42795 every 8 connection(s)
I20260929 01:52:15.127801 11503 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:52:15.134948 11585 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:52:15.136756 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 11503
I20260929 01:52:15.136931 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:52:15.139204 11585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff: Bootstrap starting.
I20260929 01:52:15.160948 11585 log.cc:826] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff: Log is configured to *not* fsync() on all Append() calls
I20260929 01:52:15.170862 11585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:52:15.171403 11585 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff: Bootstrap complete.
I20260929 01:52:15.177646 11585 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } }
I20260929 01:52:15.178308 11585 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ac72709b50c4e7588bebeae0e3cceff, State: Initialized, Role: FOLLOWER
I20260929 01:52:15.178933 11585 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } }
I20260929 01:52:15.179673 11590 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:52:15.179914 11590 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:15.180828 11585 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:52:15.191128 11601 catalog_manager.cc:1308] Loaded cluster ID: 63f4e7e6fa36411ea78e33545a86e63f
I20260929 01:52:15.191254 11601 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff: loading cluster ID for follower catalog manager: success
I20260929 01:52:15.196482 11601 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff: acquiring CA information for follower catalog manager: success
I20260929 01:52:15.199194 11601 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:52:15.216315 11503 master_runner.cc:264] No leader master found from master 9ac72709b50c4e7588bebeae0e3cceff
I20260929 01:52:15.216501 11503 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:52:15.636968 11604 raft_consensus.cc:493] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:52:15.637298 11604 raft_consensus.cc:515] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } }
I20260929 01:52:15.639267 11604 leader_election.cc:290] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers fd52acb9c5384217a5983864fb4140ab (127.0.71.125:46041), 9ac72709b50c4e7588bebeae0e3cceff (127.0.71.124:42795)
I20260929 01:52:15.646325 11476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fd52acb9c5384217a5983864fb4140ab" is_pre_election: true
I20260929 01:52:15.646988 11476 raft_consensus.cc:2468] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee9c34aa5d8d476aaf1c0daadc8c59d6 in term 1.
I20260929 01:52:15.646857 11560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9ac72709b50c4e7588bebeae0e3cceff" is_pre_election: true
I20260929 01:52:15.647405 11560 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ee9c34aa5d8d476aaf1c0daadc8c59d6 in term 1.
I20260929 01:52:15.647648 11369 leader_election.cc:304] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: ee9c34aa5d8d476aaf1c0daadc8c59d6, fd52acb9c5384217a5983864fb4140ab; no voters:
I20260929 01:52:15.648080 11604 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 01:52:15.648265 11604 raft_consensus.cc:493] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:52:15.648370 11604 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:52:15.653635 11604 raft_consensus.cc:515] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } }
I20260929 01:52:15.654392 11604 leader_election.cc:290] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [CANDIDATE]: Term 2 election: Requested vote from peers fd52acb9c5384217a5983864fb4140ab (127.0.71.125:46041), 9ac72709b50c4e7588bebeae0e3cceff (127.0.71.124:42795)
I20260929 01:52:15.654958 11560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "9ac72709b50c4e7588bebeae0e3cceff"
I20260929 01:52:15.654978 11476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "fd52acb9c5384217a5983864fb4140ab"
I20260929 01:52:15.655155 11560 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:52:15.655176 11476 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:52:15.658946 11560 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee9c34aa5d8d476aaf1c0daadc8c59d6 in term 2.
I20260929 01:52:15.659399 11369 leader_election.cc:304] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9ac72709b50c4e7588bebeae0e3cceff, ee9c34aa5d8d476aaf1c0daadc8c59d6; no voters:
I20260929 01:52:15.659754 11604 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:52:15.660116 11476 raft_consensus.cc:2468] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ee9c34aa5d8d476aaf1c0daadc8c59d6 in term 2.
I20260929 01:52:15.660225 11604 raft_consensus.cc:697] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [term 2 LEADER]: Becoming Leader. State: Replica: ee9c34aa5d8d476aaf1c0daadc8c59d6, State: Running, Role: LEADER
I20260929 01:52:15.660774 11604 consensus_queue.cc:237] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } }
I20260929 01:52:15.663218 11607 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee9c34aa5d8d476aaf1c0daadc8c59d6. Latest consensus state: current_term: 2 leader_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:52:15.663489 11607 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [sys.catalog]: This master's current role is: LEADER
I20260929 01:52:15.664194 11609 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:52:15.666107 11609 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:52:15.666878 11609 catalog_manager.cc:1308] Loaded cluster ID: 63f4e7e6fa36411ea78e33545a86e63f
I20260929 01:52:15.666976 11609 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:52:15.668052 11609 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:52:15.668877 11609 catalog_manager.cc:6230] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: Loaded TSK: 0
I20260929 01:52:15.669749 11609 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:52:15.691256 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--builtin_ntp_servers=127.0.71.84:45381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260929 01:52:16.031220 11610 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:16.031633 11610 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:16.031798 11610 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:16.040956 11610 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:16.041167 11610 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:52:16.051509 11610 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:45381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:16.053332 11610 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:16.055195 11610 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:16.065418 11615 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:16.065456 11616 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:16.066020 11618 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:16.066773 11610 server_base.cc:1061] running on GCE node
I20260929 01:52:16.067395 11610 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:16.068414 11610 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:16.069689 11610 hybrid_clock.cc:648] HybridClock initialized: now 1790646736069639 us; error 40 us; skew 500 ppm
I20260929 01:52:16.070036 11610 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:16.072192 11610 webserver.cc:533] Webserver started at http://127.0.71.65:41401/ using document root <none> and password file <none>
I20260929 01:52:16.073004 11610 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:16.073139 11610 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:16.078895 11610 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 01:52:16.082582 11624 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:16.083976 11610 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:52:16.084185 11610 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "e2615c14e9644ce1a653d800426a3c94"
format_stamp: "Formatted at 2026-09-29 01:52:00 on dist-test-slave-vz23"
I20260929 01:52:16.085033 11610 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:16.113603 11610 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:16.114508 11610 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:16.114806 11610 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:16.115813 11610 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:52:16.117544 11610 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:52:16.117652 11610 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:16.117808 11610 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:52:16.117880 11610 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:16.153618 11560 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [term 2 FOLLOWER]: Refusing update from remote peer ee9c34aa5d8d476aaf1c0daadc8c59d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 01:52:16.154711 11607 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:52:16.155376 11610 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:33059
I20260929 01:52:16.155398 11736 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:33059 every 8 connection(s)
I20260929 01:52:16.157272 11610 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:52:16.157439 11737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee9c34aa5d8d476aaf1c0daadc8c59d6. Latest consensus state: current_term: 2 leader_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:52:16.157626 11737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:16.161008 11737 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:52:16.161201 11737 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:16.162212 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 11610
I20260929 01:52:16.162480 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
I20260929 01:52:16.187258 11476 raft_consensus.cc:1275] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [term 2 FOLLOWER]: Refusing update from remote peer ee9c34aa5d8d476aaf1c0daadc8c59d6: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 01:52:16.188184 11604 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:52:16.194029 11604 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:52:16.194270 11604 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [sys.catalog]: This master's current role is: LEADER
I20260929 01:52:16.194612 11740 heartbeater.cc:344] Connected to a master server at 127.0.71.126:44069
I20260929 01:52:16.195113 11740 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:16.196327 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.66:0
--local_ip_for_outbound_sockets=127.0.71.66
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--builtin_ntp_servers=127.0.71.84:45381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:52:16.196537 11740 heartbeater.cc:507] Master 127.0.71.126:44069 requested a full tablet report, sending...
I20260929 01:52:16.199582 11382 ts_manager.cc:194] Registered new tserver with Master: e2615c14e9644ce1a653d800426a3c94 (127.0.71.65:33059)
I20260929 01:52:16.202663 11382 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:34885
I20260929 01:52:16.207974 11739 heartbeater.cc:344] Connected to a master server at 127.0.71.124:42795
I20260929 01:52:16.208117 11739 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:16.208503 11739 heartbeater.cc:507] Master 127.0.71.124:42795 requested a full tablet report, sending...
I20260929 01:52:16.210284 11550 ts_manager.cc:194] Registered new tserver with Master: e2615c14e9644ce1a653d800426a3c94 (127.0.71.65:33059)
I20260929 01:52:16.215018 11741 heartbeater.cc:344] Connected to a master server at 127.0.71.125:46041
I20260929 01:52:16.215166 11741 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:16.215441 11741 heartbeater.cc:507] Master 127.0.71.125:46041 requested a full tablet report, sending...
I20260929 01:52:16.217397 11756 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:52:16.217752 11756 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:16.218324 11751 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader ee9c34aa5d8d476aaf1c0daadc8c59d6. Latest consensus state: current_term: 2 leader_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:52:16.218549 11751 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:16.219201 11466 ts_manager.cc:194] Registered new tserver with Master: e2615c14e9644ce1a653d800426a3c94 (127.0.71.65:33059)
W20260929 01:52:16.539525 11753 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:16.539889 11753 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:16.540026 11753 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:16.548317 11753 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:16.548573 11753 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.66
I20260929 01:52:16.558772 11753 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:45381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:16.560377 11753 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:16.562140 11753 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:16.571573 11762 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:16.571750 11761 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:16.572256 11764 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:16.573180 11753 server_base.cc:1061] running on GCE node
I20260929 01:52:16.573735 11753 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:16.574717 11753 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:16.576018 11753 hybrid_clock.cc:648] HybridClock initialized: now 1790646736575970 us; error 40 us; skew 500 ppm
I20260929 01:52:16.576363 11753 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:16.578655 11753 webserver.cc:533] Webserver started at http://127.0.71.66:34695/ using document root <none> and password file <none>
I20260929 01:52:16.579365 11753 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:16.579524 11753 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:16.585976 11753 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 01:52:16.588960 11770 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:16.590477 11753 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 01:52:16.590698 11753 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/wal
uuid: "1e05a693a585490cb0fe904ba5e7c35e"
format_stamp: "Formatted at 2026-09-29 01:52:00 on dist-test-slave-vz23"
I20260929 01:52:16.591462 11753 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:16.612744 11753 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:16.613610 11753 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:16.613894 11753 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:16.614820 11753 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:52:16.616641 11753 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:52:16.616743 11753 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:16.616894 11753 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:52:16.616967 11753 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:16.654260 11753 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.66:37105
I20260929 01:52:16.654291 11882 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.66:37105 every 8 connection(s)
I20260929 01:52:16.655915 11753 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
I20260929 01:52:16.659570 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 11753
I20260929 01:52:16.659786 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-1/wal/instance
I20260929 01:52:16.663607 11604 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:52:16.663902 11604 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [sys.catalog]: This master's current role is: LEADER
I20260929 01:52:16.668059 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.67:0
--local_ip_for_outbound_sockets=127.0.71.67
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--builtin_ntp_servers=127.0.71.84:45381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:52:16.692611 11883 heartbeater.cc:344] Connected to a master server at 127.0.71.124:42795
I20260929 01:52:16.692652 11885 heartbeater.cc:344] Connected to a master server at 127.0.71.125:46041
I20260929 01:52:16.693120 11885 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:16.693240 11883 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:16.694422 11883 heartbeater.cc:507] Master 127.0.71.124:42795 requested a full tablet report, sending...
I20260929 01:52:16.694468 11885 heartbeater.cc:507] Master 127.0.71.125:46041 requested a full tablet report, sending...
I20260929 01:52:16.695647 11884 heartbeater.cc:344] Connected to a master server at 127.0.71.126:44069
I20260929 01:52:16.695787 11884 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:16.696233 11884 heartbeater.cc:507] Master 127.0.71.126:44069 requested a full tablet report, sending...
I20260929 01:52:16.696393 11466 ts_manager.cc:194] Registered new tserver with Master: 1e05a693a585490cb0fe904ba5e7c35e (127.0.71.66:37105)
I20260929 01:52:16.696727 11550 ts_manager.cc:194] Registered new tserver with Master: 1e05a693a585490cb0fe904ba5e7c35e (127.0.71.66:37105)
I20260929 01:52:16.697352 11382 ts_manager.cc:194] Registered new tserver with Master: 1e05a693a585490cb0fe904ba5e7c35e (127.0.71.66:37105)
I20260929 01:52:16.698187 11382 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.66:46577
W20260929 01:52:17.033278 11888 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:17.033591 11888 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:17.033767 11888 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:17.042289 11888 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:17.042510 11888 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.67
I20260929 01:52:17.052596 11888 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:45381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:17.054296 11888 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:17.056161 11888 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:17.066311 11895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:17.066274 11896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:17.067200 11898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:17.067608 11888 server_base.cc:1061] running on GCE node
I20260929 01:52:17.068243 11888 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:17.069262 11888 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:17.070503 11888 hybrid_clock.cc:648] HybridClock initialized: now 1790646737070460 us; error 62 us; skew 500 ppm
I20260929 01:52:17.070849 11888 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:17.073215 11888 webserver.cc:533] Webserver started at http://127.0.71.67:38543/ using document root <none> and password file <none>
I20260929 01:52:17.073988 11888 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:17.074148 11888 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:17.079610 11888 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:52:17.083297 11904 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:17.085490 11888 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 01:52:17.085759 11888 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/wal
uuid: "b76557c513e346578fbdae008c861b48"
format_stamp: "Formatted at 2026-09-29 01:52:01 on dist-test-slave-vz23"
I20260929 01:52:17.086992 11888 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:17.119661 11888 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:17.120826 11888 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:17.121212 11888 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:17.122963 11888 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:52:17.124818 11888 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:52:17.124897 11888 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:17.124989 11888 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:52:17.125084 11888 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:17.177369 11888 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.67:34463
I20260929 01:52:17.177459 12016 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.67:34463 every 8 connection(s)
I20260929 01:52:17.179108 11888 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
I20260929 01:52:17.185259 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 11888
I20260929 01:52:17.185496 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-2/wal/instance
I20260929 01:52:17.197516 12018 heartbeater.cc:344] Connected to a master server at 127.0.71.126:44069
I20260929 01:52:17.197898 12018 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:17.199000 12018 heartbeater.cc:507] Master 127.0.71.126:44069 requested a full tablet report, sending...
I20260929 01:52:17.199970 12019 heartbeater.cc:344] Connected to a master server at 127.0.71.125:46041
I20260929 01:52:17.200117 12019 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:17.200641 12019 heartbeater.cc:507] Master 127.0.71.125:46041 requested a full tablet report, sending...
I20260929 01:52:17.201103 11382 ts_manager.cc:194] Registered new tserver with Master: b76557c513e346578fbdae008c861b48 (127.0.71.67:34463)
I20260929 01:52:17.201884 11466 ts_manager.cc:194] Registered new tserver with Master: b76557c513e346578fbdae008c861b48 (127.0.71.67:34463)
I20260929 01:52:17.202209 11382 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.67:55843
I20260929 01:52:17.203255 12017 heartbeater.cc:344] Connected to a master server at 127.0.71.124:42795
I20260929 01:52:17.203389 12017 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:17.203917 12017 heartbeater.cc:507] Master 127.0.71.124:42795 requested a full tablet report, sending...
I20260929 01:52:17.204900 11550 ts_manager.cc:194] Registered new tserver with Master: b76557c513e346578fbdae008c861b48 (127.0.71.67:34463)
I20260929 01:52:17.207047 11740 heartbeater.cc:499] Master 127.0.71.126:44069 was elected leader, sending a full tablet report...
I20260929 01:52:17.211412 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:52:17.219126 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:52:17.219274 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 11351
I20260929 01:52:17.220137 11412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:17.329840 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 11351
W20260929 01:52:17.349638 11450 connection.cc:573] server connection from 127.0.0.1:42740 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:52:17.350476 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:44069
--webserver_interface=127.0.71.126
--webserver_port=46033
--builtin_ntp_servers=127.0.71.84:45381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795 with env {}
W20260929 01:52:17.685806 12025 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:17.686178 12025 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:17.686277 12025 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:17.695138 12025 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:52:17.695233 12025 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:17.695329 12025 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:52:17.695412 12025 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:52:17.700198 11884 heartbeater.cc:499] Master 127.0.71.126:44069 was elected leader, sending a full tablet report...
W20260929 01:52:17.701692 11884 heartbeater.cc:646] Failed to heartbeat to 127.0.71.126:44069 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.71.126:44069: connect: Connection refused (error 111)
I20260929 01:52:17.706871 12025 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:45381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:44069
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=46033
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:17.708527 12025 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:17.710319 12025 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:17.718436 12032 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:17.718436 12031 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:17.718930 12034 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:17.720160 12025 server_base.cc:1061] running on GCE node
I20260929 01:52:17.720964 12025 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:17.722278 12025 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:17.723474 12025 hybrid_clock.cc:648] HybridClock initialized: now 1790646737723415 us; error 53 us; skew 500 ppm
I20260929 01:52:17.723773 12025 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:17.725838 12025 webserver.cc:533] Webserver started at http://127.0.71.126:46033/ using document root <none> and password file <none>
I20260929 01:52:17.726514 12025 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:17.726599 12025 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:17.732084 12025 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 01:52:17.734966 12040 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:17.736606 12025 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:52:17.736804 12025 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6"
format_stamp: "Formatted at 2026-09-29 01:51:57 on dist-test-slave-vz23"
I20260929 01:52:17.737566 12025 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:17.772593 12025 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:17.773488 12025 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:17.773792 12025 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:17.794775 12025 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:44069
I20260929 01:52:17.794811 12091 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:44069 every 8 connection(s)
I20260929 01:52:17.796627 12025 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:52:17.800915 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 12025
I20260929 01:52:17.801221 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 11610
I20260929 01:52:17.804527 12092 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:52:17.808825 12092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: Bootstrap starting.
I20260929 01:52:17.810700 11732 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:17.826241 12092 log.cc:826] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: Log is configured to *not* fsync() on all Append() calls
I20260929 01:52:17.835314 12092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:52:17.835877 12092 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: Bootstrap complete.
I20260929 01:52:17.842569 12092 raft_consensus.cc:359] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } }
I20260929 01:52:17.843427 12092 raft_consensus.cc:740] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ee9c34aa5d8d476aaf1c0daadc8c59d6, State: Initialized, Role: FOLLOWER
I20260929 01:52:17.844269 12092 consensus_queue.cc:260] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } }
I20260929 01:52:17.845170 12098 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:52:17.845464 12098 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:17.846333 12092 sys_catalog.cc:565] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:52:17.857358 12109 catalog_manager.cc:1308] Loaded cluster ID: 63f4e7e6fa36411ea78e33545a86e63f
I20260929 01:52:17.857501 12109 catalog_manager.cc:1601] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: loading cluster ID for follower catalog manager: success
I20260929 01:52:17.863358 12109 catalog_manager.cc:1623] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: acquiring CA information for follower catalog manager: success
I20260929 01:52:17.867038 12109 catalog_manager.cc:1651] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:52:17.902014 12025 master_runner.cc:264] No leader master found from master ee9c34aa5d8d476aaf1c0daadc8c59d6
I20260929 01:52:17.902181 12025 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:52:17.916833 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 11610
I20260929 01:52:17.933375 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:33059
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--webserver_port=41401
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:45381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:52:18.204766 12018 heartbeater.cc:499] Master 127.0.71.126:44069 was elected leader, sending a full tablet report...
I20260929 01:52:18.215540 12057 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "b76557c513e346578fbdae008c861b48" instance_seqno: 1790646737166835) as {username='slave'} at 127.0.71.67:56863; Asking this server to re-register.
I20260929 01:52:18.216385 12018 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:18.218034 12057 ts_manager.cc:194] Registered new tserver with Master: b76557c513e346578fbdae008c861b48 (127.0.71.67:34463)
W20260929 01:52:18.274520 12115 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:18.274900 12115 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:18.275040 12115 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:18.284374 12115 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:18.284653 12115 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:52:18.295301 12115 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:45381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:33059
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=41401
--tserver_master_addrs=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:18.297129 12115 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:18.299026 12115 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:18.309095 12122 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:18.309612 12121 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:18.310253 12124 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:18.310421 12115 server_base.cc:1061] running on GCE node
I20260929 01:52:18.310978 12115 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:18.311939 12115 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:18.313198 12115 hybrid_clock.cc:648] HybridClock initialized: now 1790646738313133 us; error 45 us; skew 500 ppm
I20260929 01:52:18.313519 12115 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:18.315692 12115 webserver.cc:533] Webserver started at http://127.0.71.65:41401/ using document root <none> and password file <none>
I20260929 01:52:18.316553 12115 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:18.316712 12115 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:18.322196 12115 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:52:18.325069 12130 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:18.326432 12115 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:52:18.326622 12115 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "e2615c14e9644ce1a653d800426a3c94"
format_stamp: "Formatted at 2026-09-29 01:52:00 on dist-test-slave-vz23"
I20260929 01:52:18.327338 12115 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:18.357645 12115 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:18.358534 12115 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:18.358816 12115 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:18.359742 12115 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:52:18.361486 12115 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:52:18.361593 12115 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:18.361776 12115 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:52:18.361852 12115 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:18.397441 12115 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:33059
I20260929 01:52:18.397502 12242 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:33059 every 8 connection(s)
I20260929 01:52:18.399065 12115 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:52:18.404394 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 12115
I20260929 01:52:18.415428 12243 heartbeater.cc:344] Connected to a master server at 127.0.71.124:42795
I20260929 01:52:18.415841 12243 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:18.417016 12243 heartbeater.cc:507] Master 127.0.71.124:42795 requested a full tablet report, sending...
I20260929 01:52:18.417273 12245 heartbeater.cc:344] Connected to a master server at 127.0.71.125:46041
I20260929 01:52:18.417425 12245 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:18.417984 12245 heartbeater.cc:507] Master 127.0.71.125:46041 requested a full tablet report, sending...
I20260929 01:52:18.419148 11466 ts_manager.cc:194] Re-registered known tserver with Master: e2615c14e9644ce1a653d800426a3c94 (127.0.71.65:33059)
I20260929 01:52:18.419179 11550 ts_manager.cc:194] Re-registered known tserver with Master: e2615c14e9644ce1a653d800426a3c94 (127.0.71.65:33059)
I20260929 01:52:18.419459 12244 heartbeater.cc:344] Connected to a master server at 127.0.71.126:44069
I20260929 01:52:18.419605 12244 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:18.420100 12244 heartbeater.cc:507] Master 127.0.71.126:44069 requested a full tablet report, sending...
I20260929 01:52:18.421231 12057 ts_manager.cc:194] Registered new tserver with Master: e2615c14e9644ce1a653d800426a3c94 (127.0.71.65:33059)
I20260929 01:52:18.507247 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 12115
I20260929 01:52:18.516795 12238 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:18.606061 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 12115
I20260929 01:52:18.623250 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:33059
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--webserver_port=41401
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:45381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {}
I20260929 01:52:18.628351 12257 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee9c34aa5d8d476aaf1c0daadc8c59d6)
I20260929 01:52:18.628630 12257 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } }
I20260929 01:52:18.630676 12257 leader_election.cc:290] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee9c34aa5d8d476aaf1c0daadc8c59d6 (127.0.71.126:44069), fd52acb9c5384217a5983864fb4140ab (127.0.71.125:46041)
I20260929 01:52:18.631337 11476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ac72709b50c4e7588bebeae0e3cceff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fd52acb9c5384217a5983864fb4140ab" is_pre_election: true
I20260929 01:52:18.637432 12067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ac72709b50c4e7588bebeae0e3cceff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" is_pre_election: true
I20260929 01:52:18.637928 12067 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9ac72709b50c4e7588bebeae0e3cceff in term 2.
I20260929 01:52:18.638533 11536 leader_election.cc:304] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ac72709b50c4e7588bebeae0e3cceff, ee9c34aa5d8d476aaf1c0daadc8c59d6; no voters: fd52acb9c5384217a5983864fb4140ab
I20260929 01:52:18.639017 12257 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 01:52:18.639158 12257 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [term 2 FOLLOWER]: Starting leader election (detected failure of leader ee9c34aa5d8d476aaf1c0daadc8c59d6)
I20260929 01:52:18.639287 12257 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:52:18.641505 12257 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } }
I20260929 01:52:18.642148 12257 leader_election.cc:290] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [CANDIDATE]: Term 3 election: Requested vote from peers ee9c34aa5d8d476aaf1c0daadc8c59d6 (127.0.71.126:44069), fd52acb9c5384217a5983864fb4140ab (127.0.71.125:46041)
I20260929 01:52:18.642689 11476 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ac72709b50c4e7588bebeae0e3cceff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "fd52acb9c5384217a5983864fb4140ab"
I20260929 01:52:18.642714 12067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9ac72709b50c4e7588bebeae0e3cceff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6"
I20260929 01:52:18.642951 12067 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:52:18.648497 12067 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9ac72709b50c4e7588bebeae0e3cceff in term 3.
I20260929 01:52:18.648981 11536 leader_election.cc:304] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 9ac72709b50c4e7588bebeae0e3cceff, ee9c34aa5d8d476aaf1c0daadc8c59d6; no voters: fd52acb9c5384217a5983864fb4140ab
I20260929 01:52:18.649371 12257 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [term 3 FOLLOWER]: Leader election won for term 3
I20260929 01:52:18.649881 12257 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [term 3 LEADER]: Becoming Leader. State: Replica: 9ac72709b50c4e7588bebeae0e3cceff, State: Running, Role: LEADER
I20260929 01:52:18.650418 12257 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } }
I20260929 01:52:18.653012 12259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ac72709b50c4e7588bebeae0e3cceff. Latest consensus state: current_term: 3 leader_uuid: "9ac72709b50c4e7588bebeae0e3cceff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:52:18.653267 12259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [sys.catalog]: This master's current role is: LEADER
I20260929 01:52:18.661001 12261 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:52:18.663573 12261 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:52:18.664600 12261 catalog_manager.cc:1308] Loaded cluster ID: 63f4e7e6fa36411ea78e33545a86e63f
I20260929 01:52:18.664690 12261 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:52:18.666121 12261 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:52:18.667129 12261 catalog_manager.cc:6230] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff: Loaded TSK: 0
I20260929 01:52:18.668072 12261 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:52:18.708737 11884 heartbeater.cc:344] Connected to a master server at 127.0.71.126:44069
I20260929 01:52:18.709012 11884 heartbeater.cc:499] Master 127.0.71.126:44069 was elected leader, sending a full tablet report...
I20260929 01:52:18.710047 12057 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "1e05a693a585490cb0fe904ba5e7c35e" instance_seqno: 1790646736643464) as {username='slave'} at 127.0.71.66:33537; Asking this server to re-register.
I20260929 01:52:18.710742 11884 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:18.711979 12057 ts_manager.cc:194] Registered new tserver with Master: 1e05a693a585490cb0fe904ba5e7c35e (127.0.71.66:37105)
I20260929 01:52:18.831708 12263 raft_consensus.cc:493] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ee9c34aa5d8d476aaf1c0daadc8c59d6)
I20260929 01:52:18.831960 12263 raft_consensus.cc:515] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } }
I20260929 01:52:18.833922 12263 leader_election.cc:290] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ee9c34aa5d8d476aaf1c0daadc8c59d6 (127.0.71.126:44069), 9ac72709b50c4e7588bebeae0e3cceff (127.0.71.124:42795)
I20260929 01:52:18.842530 11560 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd52acb9c5384217a5983864fb4140ab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "9ac72709b50c4e7588bebeae0e3cceff" is_pre_election: true
I20260929 01:52:18.843667 12067 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "fd52acb9c5384217a5983864fb4140ab" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" is_pre_election: true
I20260929 01:52:18.844007 12067 raft_consensus.cc:2393] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate fd52acb9c5384217a5983864fb4140ab in current term 3: Already voted for candidate 9ac72709b50c4e7588bebeae0e3cceff in this term.
I20260929 01:52:18.844729 11452 leader_election.cc:304] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: fd52acb9c5384217a5983864fb4140ab; no voters: 9ac72709b50c4e7588bebeae0e3cceff, ee9c34aa5d8d476aaf1c0daadc8c59d6
I20260929 01:52:18.845156 12263 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:52:18.848400 12263 raft_consensus.cc:2749] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260929 01:52:18.971007 12256 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:18.971297 12256 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:18.971369 12256 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:52:18.971426 12256 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:18.979832 12256 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:18.980026 12256 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:52:18.990190 12256 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:45381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:33059
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=41401
--tserver_master_addrs=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:18.991757 12256 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:18.993629 12256 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:19.003692 12271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:19.003713 12270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:19.005241 12273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:19.006114 12256 server_base.cc:1061] running on GCE node
I20260929 01:52:19.006891 12256 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:19.008159 12256 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:19.009405 12256 hybrid_clock.cc:648] HybridClock initialized: now 1790646739009320 us; error 49 us; skew 500 ppm
I20260929 01:52:19.009756 12256 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:19.012763 12256 webserver.cc:533] Webserver started at http://127.0.71.65:41401/ using document root <none> and password file <none>
I20260929 01:52:19.013729 12256 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:19.013891 12256 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:19.021893 12256 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 01:52:19.024948 12279 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:19.026405 12256 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:52:19.026594 12256 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "e2615c14e9644ce1a653d800426a3c94"
format_stamp: "Formatted at 2026-09-29 01:52:00 on dist-test-slave-vz23"
I20260929 01:52:19.027305 12256 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:19.050452 12256 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:19.051342 12256 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:19.051625 12256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:19.052690 12256 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:52:19.054499 12256 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:52:19.054623 12256 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:19.054768 12256 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:52:19.054854 12256 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:52:19.055333 12256 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:52:19.055450 12256 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:52:19.055532 12256 fault_injection.cc:59] injecting fault for kudu (pid 12256): FLAGS_fault_before_start (process will exit)
I20260929 01:52:19.074666 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:33059
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--webserver_port=41401
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:45381
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:52:19.115743 12067 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [term 3 FOLLOWER]: Refusing update from remote peer 9ac72709b50c4e7588bebeae0e3cceff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 01:52:19.116760 12259 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [LEADER]: Connected to new peer: Peer: permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:52:19.120554 12288 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ac72709b50c4e7588bebeae0e3cceff. Latest consensus state: current_term: 3 leader_uuid: "9ac72709b50c4e7588bebeae0e3cceff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:52:19.120862 12288 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:19.122081 12257 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "9ac72709b50c4e7588bebeae0e3cceff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:52:19.122414 12257 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [sys.catalog]: This master's current role is: LEADER
I20260929 01:52:19.123530 11476 raft_consensus.cc:1275] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [term 3 FOLLOWER]: Refusing update from remote peer 9ac72709b50c4e7588bebeae0e3cceff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 01:52:19.124395 12257 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:52:19.135200 12288 sys_catalog.cc:455] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "9ac72709b50c4e7588bebeae0e3cceff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:52:19.135437 12288 sys_catalog.cc:458] T 00000000000000000000000000000000 P ee9c34aa5d8d476aaf1c0daadc8c59d6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:19.141472 12263 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9ac72709b50c4e7588bebeae0e3cceff. Latest consensus state: current_term: 3 leader_uuid: "9ac72709b50c4e7588bebeae0e3cceff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:52:19.141425 12295 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "9ac72709b50c4e7588bebeae0e3cceff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:52:19.141724 12295 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:19.141729 12263 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd52acb9c5384217a5983864fb4140ab [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:52:19.421010 12287 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:19.421392 12287 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:19.421531 12287 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:19.430120 12287 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:19.430362 12287 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:52:19.440346 12287 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:45381
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:33059
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=41401
--tserver_master_addrs=127.0.71.126:44069,127.0.71.125:46041,127.0.71.124:42795
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:19.442102 12287 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:19.443954 12287 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:19.454456 12300 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:19.454784 12301 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:19.454823 12303 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:19.455576 12287 server_base.cc:1061] running on GCE node
I20260929 01:52:19.456182 12287 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:19.457369 12287 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:19.458606 12287 hybrid_clock.cc:648] HybridClock initialized: now 1790646739458506 us; error 72 us; skew 500 ppm
I20260929 01:52:19.458936 12287 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:19.461148 12287 webserver.cc:533] Webserver started at http://127.0.71.65:41401/ using document root <none> and password file <none>
I20260929 01:52:19.461879 12287 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:19.462016 12287 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:19.467511 12287 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:52:19.470293 12309 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:19.471706 12287 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:52:19.471884 12287 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "e2615c14e9644ce1a653d800426a3c94"
format_stamp: "Formatted at 2026-09-29 01:52:00 on dist-test-slave-vz23"
I20260929 01:52:19.472644 12287 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:19.491688 12287 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:19.492565 12287 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:19.492854 12287 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:19.493880 12287 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:52:19.495558 12287 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:52:19.495659 12287 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:19.495808 12287 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:52:19.495878 12287 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:19.530634 12287 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:33059
I20260929 01:52:19.530675 12421 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:33059 every 8 connection(s)
I20260929 01:52:19.532086 12287 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:52:19.535066 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 12287
I20260929 01:52:19.535401 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 12287
I20260929 01:52:19.584306 12422 heartbeater.cc:344] Connected to a master server at 127.0.71.124:42795
I20260929 01:52:19.584376 12424 heartbeater.cc:344] Connected to a master server at 127.0.71.125:46041
I20260929 01:52:19.584759 12422 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:19.584765 12424 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:19.584255 12423 heartbeater.cc:344] Connected to a master server at 127.0.71.126:44069
I20260929 01:52:19.585007 12423 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:19.585110 12417 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:19.631078 12259 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "9ac72709b50c4e7588bebeae0e3cceff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ee9c34aa5d8d476aaf1c0daadc8c59d6" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44069 } } peers { permanent_uuid: "fd52acb9c5384217a5983864fb4140ab" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46041 } } peers { permanent_uuid: "9ac72709b50c4e7588bebeae0e3cceff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 42795 } } }
I20260929 01:52:19.631363 12259 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff [sys.catalog]: This master's current role is: LEADER
I20260929 01:52:19.676265 12423 heartbeater.cc:507] Master 127.0.71.126:44069 requested a full tablet report, sending...
I20260929 01:52:19.676294 12422 heartbeater.cc:507] Master 127.0.71.124:42795 requested a full tablet report, sending...
I20260929 01:52:19.676645 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 12287
I20260929 01:52:19.692907 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 11753
I20260929 01:52:19.701284 11878 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:19.702374 11883 heartbeater.cc:499] Master 127.0.71.124:42795 was elected leader, sending a full tablet report...
I20260929 01:52:19.789774 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 11753
I20260929 01:52:19.806473 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 11888
I20260929 01:52:19.814059 12012 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:19.906123 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 11888
I20260929 01:52:19.922149 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 12025
I20260929 01:52:19.928360 12087 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:20.033434 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 12025
I20260929 01:52:20.052893 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 11419
I20260929 01:52:20.054042 11496 generic_service.cc:196] Checking for leaks (request via RPC)
W20260929 01:52:20.119753 11536 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.126:44069: connect: Connection refused (error 111)
W20260929 01:52:20.123399 11536 consensus_peers.cc:597] T 00000000000000000000000000000000 P 9ac72709b50c4e7588bebeae0e3cceff -> Peer ee9c34aa5d8d476aaf1c0daadc8c59d6 (127.0.71.126:44069): Couldn't send request to peer ee9c34aa5d8d476aaf1c0daadc8c59d6. Status: Network error: Client connection negotiation failed: client connection to 127.0.71.126:44069: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 01:52:20.168082 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 11419
I20260929 01:52:20.187777 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 11503
I20260929 01:52:20.189040 11580 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:20.298187 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 11503
Shutting down hive metastore.
2026-09-29T01:52:20Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestBasicOperation/4 (39028 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/5
2026-09-29T01:52:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:52:20Z Disabled control of system clock
2026-09-29T01:52:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:52:20Z Disabled control of system clock
2026-09-29T01:52:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:52:20Z Disabled control of system clock
2026-09-29T01:52:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:52:20Z Disabled control of system clock
2026-09-29T01:52:20Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:52:20Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/security.properties
2026-09-29 01:52:28: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260929 01:52:36.699863 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:42257
--webserver_interface=127.0.71.126
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:35469,127.0.71.85:39963,127.0.71.86:37273,127.0.71.87:41751,127.0.71.88:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873 with env {}
W20260929 01:52:37.025650 12735 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:37.025983 12735 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:37.026108 12735 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:37.034500 12735 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:52:37.034605 12735 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:37.034691 12735 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:52:37.034780 12735 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:52:37.045421 12735 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35469,127.0.71.85:39963,127.0.71.86:37273,127.0.71.87:41751,127.0.71.88:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:42257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:37.047096 12735 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:37.048996 12735 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:37.057313 12740 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:37.057351 12741 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:37.057833 12743 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:37.058183 12735 server_base.cc:1061] running on GCE node
I20260929 01:52:37.058895 12735 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:37.060429 12735 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:37.061710 12735 hybrid_clock.cc:648] HybridClock initialized: now 1790646757061591 us; error 88 us; skew 500 ppm
I20260929 01:52:37.062093 12735 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:37.064162 12735 webserver.cc:533] Webserver started at http://127.0.71.126:45621/ using document root <none> and password file <none>
I20260929 01:52:37.064926 12735 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:37.065088 12735 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:37.065461 12735 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:52:37.067778 12735 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "44f4988489a442e989d3e74837e674ad"
format_stamp: "Formatted at 2026-09-29 01:52:37 on dist-test-slave-vz23"
I20260929 01:52:37.068599 12735 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "44f4988489a442e989d3e74837e674ad"
format_stamp: "Formatted at 2026-09-29 01:52:37 on dist-test-slave-vz23"
I20260929 01:52:37.073909 12735 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 01:52:37.077488 12749 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:37.078811 12735 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:52:37.078979 12735 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "44f4988489a442e989d3e74837e674ad"
format_stamp: "Formatted at 2026-09-29 01:52:37 on dist-test-slave-vz23"
I20260929 01:52:37.079200 12735 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:37.118854 12735 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:37.119848 12735 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:37.120157 12735 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:37.140884 12735 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:42257
I20260929 01:52:37.140908 12800 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:42257 every 8 connection(s)
I20260929 01:52:37.142449 12735 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:52:37.146679 12801 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:52:37.149983 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 12735
I20260929 01:52:37.150202 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:52:37.151499 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:44797
--webserver_interface=127.0.71.125
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:35469,127.0.71.85:39963,127.0.71.86:37273,127.0.71.87:41751,127.0.71.88:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873 with env {}
I20260929 01:52:37.154435 12801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:52:37.172103 12801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:52:37.173357 12751 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.125:44797: connect: Connection refused (error 111)
W20260929 01:52:37.174921 12801 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:44797: Network error: Client connection negotiation failed: client connection to 127.0.71.125:44797: connect: Connection refused (error 111)
I20260929 01:52:37.224342 12801 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } attempt: 1
W20260929 01:52:37.226781 12801 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:44797: Network error: Client connection negotiation failed: client connection to 127.0.71.125:44797: connect: Connection refused (error 111)
I20260929 01:52:37.295043 12801 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } attempt: 2
W20260929 01:52:37.297219 12801 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:44797: Network error: Client connection negotiation failed: client connection to 127.0.71.125:44797: connect: Connection refused (error 111)
I20260929 01:52:37.388543 12801 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } attempt: 3
W20260929 01:52:37.390553 12801 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:44797: Network error: Client connection negotiation failed: client connection to 127.0.71.125:44797: connect: Connection refused (error 111)
W20260929 01:52:37.484294 12803 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:37.484710 12803 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:37.484797 12803 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:37.493085 12803 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:52:37.493171 12803 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:37.493269 12803 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:52:37.493338 12803 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:52:37.503329 12803 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35469,127.0.71.85:39963,127.0.71.86:37273,127.0.71.87:41751,127.0.71.88:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:44797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:37.505018 12803 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:37.506752 12803 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:37.515170 12812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:37.515159 12811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:37.515193 12814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:37.516620 12803 server_base.cc:1061] running on GCE node
I20260929 01:52:37.517484 12803 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:37.519270 12803 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:37.520606 12803 hybrid_clock.cc:648] HybridClock initialized: now 1790646757520455 us; error 102 us; skew 500 ppm
I20260929 01:52:37.521039 12803 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:37.523536 12803 webserver.cc:533] Webserver started at http://127.0.71.125:40319/ using document root <none> and password file <none>
I20260929 01:52:37.524372 12803 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:37.524564 12803 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:37.524981 12803 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:52:37.528903 12803 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/data/instance:
uuid: "d7efd97cc2a740c0a88ee4a714d6748a"
format_stamp: "Formatted at 2026-09-29 01:52:37 on dist-test-slave-vz23"
I20260929 01:52:37.529728 12803 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/wal/instance:
uuid: "d7efd97cc2a740c0a88ee4a714d6748a"
format_stamp: "Formatted at 2026-09-29 01:52:37 on dist-test-slave-vz23"
I20260929 01:52:37.533826 12801 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } attempt: 4
I20260929 01:52:37.535526 12803 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
W20260929 01:52:37.535740 12801 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:44797: Network error: Client connection negotiation failed: client connection to 127.0.71.125:44797: connect: Connection refused (error 111)
I20260929 01:52:37.539292 12820 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:37.540845 12803 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 01:52:37.541033 12803 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "d7efd97cc2a740c0a88ee4a714d6748a"
format_stamp: "Formatted at 2026-09-29 01:52:37 on dist-test-slave-vz23"
I20260929 01:52:37.541252 12803 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:37.568643 12803 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:37.569556 12803 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:37.569849 12803 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:37.591310 12803 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:44797
I20260929 01:52:37.591329 12871 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:44797 every 8 connection(s)
I20260929 01:52:37.593011 12803 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:52:37.597280 12872 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:52:37.598309 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 12803
I20260929 01:52:37.598491 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:52:37.599877 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:40873
--webserver_interface=127.0.71.124
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:35469,127.0.71.85:39963,127.0.71.86:37273,127.0.71.87:41751,127.0.71.88:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873 with env {}
I20260929 01:52:37.606084 12872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:52:37.619292 12872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:52:37.629769 12872 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:52:37.631189 12821 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40873: connect: Connection refused (error 111)
W20260929 01:52:37.633137 12872 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:40873: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40873: connect: Connection refused (error 111)
I20260929 01:52:37.682473 12872 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } attempt: 1
W20260929 01:52:37.684633 12872 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:40873: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40873: connect: Connection refused (error 111)
I20260929 01:52:37.752907 12872 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } attempt: 2
W20260929 01:52:37.754942 12872 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:40873: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40873: connect: Connection refused (error 111)
I20260929 01:52:37.835007 12801 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } attempt: 5
I20260929 01:52:37.841857 12801 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:52:37.843907 12801 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:40873: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40873: connect: Connection refused (error 111)
I20260929 01:52:37.846202 12872 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } attempt: 3
W20260929 01:52:37.848004 12872 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:40873: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40873: connect: Connection refused (error 111)
I20260929 01:52:37.895135 12801 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } attempt: 1
W20260929 01:52:37.897198 12801 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:40873: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40873: connect: Connection refused (error 111)
W20260929 01:52:37.932497 12874 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:37.932835 12874 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:37.932952 12874 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:37.941538 12874 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:52:37.941632 12874 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:37.941726 12874 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:52:37.941809 12874 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:52:37.952059 12874 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35469,127.0.71.85:39963,127.0.71.86:37273,127.0.71.87:41751,127.0.71.88:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:40873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:37.953742 12874 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:37.955454 12874 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:37.963331 12882 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:37.963299 12883 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:37.964031 12885 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:37.964740 12874 server_base.cc:1061] running on GCE node
I20260929 01:52:37.965416 12801 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } attempt: 2
I20260929 01:52:37.965480 12874 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20260929 01:52:37.967207 12801 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:40873: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40873: connect: Connection refused (error 111)
I20260929 01:52:37.967594 12874 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:37.968811 12874 hybrid_clock.cc:648] HybridClock initialized: now 1790646757968685 us; error 99 us; skew 500 ppm
I20260929 01:52:37.969105 12874 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:37.971197 12874 webserver.cc:533] Webserver started at http://127.0.71.124:33367/ using document root <none> and password file <none>
I20260929 01:52:37.971853 12874 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:37.971937 12874 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:37.972237 12874 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:52:37.974643 12874 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/data/instance:
uuid: "ae67ac031b4c4b84a8b717d053cb563d"
format_stamp: "Formatted at 2026-09-29 01:52:37 on dist-test-slave-vz23"
I20260929 01:52:37.975338 12874 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/wal/instance:
uuid: "ae67ac031b4c4b84a8b717d053cb563d"
format_stamp: "Formatted at 2026-09-29 01:52:37 on dist-test-slave-vz23"
I20260929 01:52:37.980575 12874 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 01:52:37.983855 12891 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:37.985241 12874 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:52:37.985420 12874 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "ae67ac031b4c4b84a8b717d053cb563d"
format_stamp: "Formatted at 2026-09-29 01:52:37 on dist-test-slave-vz23"
I20260929 01:52:37.985646 12874 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:37.991227 12872 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } attempt: 4
W20260929 01:52:37.993188 12872 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:40873: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40873: connect: Connection refused (error 111)
I20260929 01:52:38.022221 12874 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:38.023167 12874 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:38.023466 12874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:38.044953 12874 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:40873
I20260929 01:52:38.044965 12942 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:40873 every 8 connection(s)
I20260929 01:52:38.046720 12874 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:52:38.051017 12943 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:52:38.051916 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 12874
I20260929 01:52:38.052124 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:52:38.058976 12943 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:52:38.073437 12801 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } attempt: 3
I20260929 01:52:38.073901 12943 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:52:38.081103 12943 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:52:38.088322 12801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: Bootstrap starting.
I20260929 01:52:38.093816 12943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d: Bootstrap starting.
I20260929 01:52:38.093930 12801 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: Neither blocks nor log segments found. Creating new log.
I20260929 01:52:38.095409 12801 log.cc:826] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: Log is configured to *not* fsync() on all Append() calls
I20260929 01:52:38.098086 12943 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d: Neither blocks nor log segments found. Creating new log.
I20260929 01:52:38.098625 12801 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: No bootstrap required, opened a new log
I20260929 01:52:38.099413 12943 log.cc:826] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:52:38.102433 12943 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d: No bootstrap required, opened a new log
I20260929 01:52:38.106812 12801 raft_consensus.cc:359] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:38.107167 12801 raft_consensus.cc:385] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:52:38.107348 12801 raft_consensus.cc:740] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f4988489a442e989d3e74837e674ad, State: Initialized, Role: FOLLOWER
I20260929 01:52:38.108090 12801 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:38.110219 12950 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:38.110688 12950 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:38.110680 12943 raft_consensus.cc:359] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:38.110977 12801 sys_catalog.cc:565] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:52:38.111104 12943 raft_consensus.cc:385] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:52:38.111276 12943 raft_consensus.cc:740] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae67ac031b4c4b84a8b717d053cb563d, State: Initialized, Role: FOLLOWER
I20260929 01:52:38.112064 12943 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:38.113315 12951 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:38.113718 12951 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:38.114670 12943 sys_catalog.cc:565] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:52:38.122898 12755 tablet.cc:2433] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:52:38.124576 12966 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:52:38.124805 12966 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 01:52:38.130090 12974 catalog_manager.cc:1607] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:52:38.130329 12974 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:52:38.123756 12735 master_runner.cc:187] Error getting master registration for 127.0.71.125:44797: OK, instance_id { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" instance_seqno: 1790646757585811 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:52:38.130728 12735 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:52:38.135957 12874 master_runner.cc:187] Error getting master registration for 127.0.71.125:44797: OK, instance_id { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" instance_seqno: 1790646757585811 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:52:38.142745 12874 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:52:38.275815 12897 tablet.cc:2433] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:52:38.292411 12872 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } attempt: 5
I20260929 01:52:38.306212 12872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a: Bootstrap starting.
I20260929 01:52:38.311594 12872 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a: Neither blocks nor log segments found. Creating new log.
I20260929 01:52:38.313330 12872 log.cc:826] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a: Log is configured to *not* fsync() on all Append() calls
I20260929 01:52:38.316774 12950 raft_consensus.cc:493] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:52:38.317087 12872 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a: No bootstrap required, opened a new log
I20260929 01:52:38.317052 12950 raft_consensus.cc:515] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:38.318513 12950 leader_election.cc:290] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d7efd97cc2a740c0a88ee4a714d6748a (127.0.71.125:44797), ae67ac031b4c4b84a8b717d053cb563d (127.0.71.124:40873)
I20260929 01:52:38.320034 12918 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44f4988489a442e989d3e74837e674ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae67ac031b4c4b84a8b717d053cb563d" is_pre_election: true
I20260929 01:52:38.320675 12918 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f4988489a442e989d3e74837e674ad in term 0.
I20260929 01:52:38.321530 12750 leader_election.cc:304] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f4988489a442e989d3e74837e674ad, ae67ac031b4c4b84a8b717d053cb563d; no voters:
I20260929 01:52:38.322014 12950 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:52:38.322175 12950 raft_consensus.cc:493] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:52:38.322292 12950 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:52:38.326030 12950 raft_consensus.cc:515] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:38.327440 12950 leader_election.cc:290] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [CANDIDATE]: Term 1 election: Requested vote from peers d7efd97cc2a740c0a88ee4a714d6748a (127.0.71.125:44797), ae67ac031b4c4b84a8b717d053cb563d (127.0.71.124:40873)
I20260929 01:52:38.327869 12918 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44f4988489a442e989d3e74837e674ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae67ac031b4c4b84a8b717d053cb563d"
I20260929 01:52:38.328088 12918 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:52:38.335415 12872 raft_consensus.cc:359] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:38.335788 12872 raft_consensus.cc:385] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:52:38.335932 12872 raft_consensus.cc:740] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7efd97cc2a740c0a88ee4a714d6748a, State: Initialized, Role: FOLLOWER
I20260929 01:52:38.336820 12872 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:38.336879 12918 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f4988489a442e989d3e74837e674ad in term 1.
I20260929 01:52:38.339702 12750 leader_election.cc:304] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f4988489a442e989d3e74837e674ad, ae67ac031b4c4b84a8b717d053cb563d; no voters:
I20260929 01:52:38.340044 12950 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:52:38.347362 12846 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44f4988489a442e989d3e74837e674ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7efd97cc2a740c0a88ee4a714d6748a"
I20260929 01:52:38.347260 12872 sys_catalog.cc:565] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:52:38.347901 12846 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:52:38.351517 12846 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f4988489a442e989d3e74837e674ad in term 1.
I20260929 01:52:38.347311 12847 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44f4988489a442e989d3e74837e674ad" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" is_pre_election: true
I20260929 01:52:38.352743 12847 raft_consensus.cc:2376] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 44f4988489a442e989d3e74837e674ad in term 1. Re-sending same reply.
I20260929 01:52:38.352614 12977 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:38.352948 12977 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:38.357463 12950 raft_consensus.cc:697] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 1 LEADER]: Becoming Leader. State: Replica: 44f4988489a442e989d3e74837e674ad, State: Running, Role: LEADER
I20260929 01:52:38.358109 12950 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:38.370914 12978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44f4988489a442e989d3e74837e674ad. Latest consensus state: current_term: 1 leader_uuid: "44f4988489a442e989d3e74837e674ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:38.371253 12978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: This master's current role is: LEADER
I20260929 01:52:38.378865 12984 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:52:38.381803 12984 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:52:38.402288 12918 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 1 FOLLOWER]: Refusing update from remote peer 44f4988489a442e989d3e74837e674ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:52:38.403668 12978 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:52:38.411339 12846 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [term 1 FOLLOWER]: Refusing update from remote peer 44f4988489a442e989d3e74837e674ad: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:52:38.412602 12978 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:52:38.441017 12977 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44f4988489a442e989d3e74837e674ad. Latest consensus state: current_term: 1 leader_uuid: "44f4988489a442e989d3e74837e674ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:38.441354 12977 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:38.441447 12951 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44f4988489a442e989d3e74837e674ad. Latest consensus state: current_term: 1 leader_uuid: "44f4988489a442e989d3e74837e674ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:38.441695 12951 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:38.445791 12987 mvcc.cc:204] Tried to move back new op lower bound from 7334489122401501184 to 7334489122240450560. Current Snapshot: MvccSnapshot[applied={T|T < 7334489122401501184}]
I20260929 01:52:38.449862 12978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "44f4988489a442e989d3e74837e674ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:38.450124 12978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: This master's current role is: LEADER
I20260929 01:52:38.450377 12978 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "44f4988489a442e989d3e74837e674ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:38.450580 12978 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: This master's current role is: LEADER
I20260929 01:52:38.451016 12993 mvcc.cc:204] Tried to move back new op lower bound from 7334489122401501184 to 7334489122240450560. Current Snapshot: MvccSnapshot[applied={T|T < 7334489122401501184}]
I20260929 01:52:38.462842 12977 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "44f4988489a442e989d3e74837e674ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:38.463061 12977 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:38.465092 12951 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "44f4988489a442e989d3e74837e674ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:38.465315 12951 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:38.471422 12984 catalog_manager.cc:1396] Generated new cluster ID: 3ed8461d403447e3800fefe76334ccef
I20260929 01:52:38.471545 12984 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
W20260929 01:52:38.479024 12996 catalog_manager.cc:1607] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:52:38.479167 12996 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:52:38.527232 12984 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 01:52:38.530375 12984 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:52:38.552223 12984 catalog_manager.cc:6219] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: Generated new TSK 0
I20260929 01:52:38.553704 12984 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:52:38.568753 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--builtin_ntp_servers=127.0.71.84:35469,127.0.71.85:39963,127.0.71.86:37273,127.0.71.87:41751,127.0.71.88:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260929 01:52:38.975162 13001 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:38.975495 13001 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:38.975629 13001 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:38.984134 13001 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:38.984395 13001 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:52:38.994789 13001 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35469,127.0.71.85:39963,127.0.71.86:37273,127.0.71.87:41751,127.0.71.88:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:38.996536 13001 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:38.998436 13001 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:39.011011 13006 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:39.012495 13007 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:39.014391 13009 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:39.015707 13001 server_base.cc:1061] running on GCE node
I20260929 01:52:39.016659 13001 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:39.018378 13001 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:39.021754 13001 hybrid_clock.cc:648] HybridClock initialized: now 1790646759021492 us; error 225 us; skew 500 ppm
I20260929 01:52:39.022179 13001 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:39.025273 13001 webserver.cc:533] Webserver started at http://127.0.71.65:34699/ using document root <none> and password file <none>
I20260929 01:52:39.026278 13001 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:39.026397 13001 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:39.026824 13001 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:52:39.030512 13001 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "f796548a8f984835bc232e10b6ad1f76"
format_stamp: "Formatted at 2026-09-29 01:52:39 on dist-test-slave-vz23"
I20260929 01:52:39.031610 13001 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "f796548a8f984835bc232e10b6ad1f76"
format_stamp: "Formatted at 2026-09-29 01:52:39 on dist-test-slave-vz23"
I20260929 01:52:39.038471 13001 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 01:52:39.042057 13015 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:39.043473 13001 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 01:52:39.043654 13001 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "f796548a8f984835bc232e10b6ad1f76"
format_stamp: "Formatted at 2026-09-29 01:52:39 on dist-test-slave-vz23"
I20260929 01:52:39.043866 13001 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:39.071781 13001 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:39.072755 13001 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:39.073032 13001 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:39.073975 13001 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:52:39.075713 13001 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:52:39.075831 13001 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:39.075956 13001 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:52:39.076038 13001 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:39.114634 13001 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:44525
I20260929 01:52:39.114732 13128 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:44525 every 8 connection(s)
I20260929 01:52:39.116604 13001 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:52:39.116945 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 13001
I20260929 01:52:39.117151 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
I20260929 01:52:39.126958 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.66:0
--local_ip_for_outbound_sockets=127.0.71.66
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--builtin_ntp_servers=127.0.71.84:35469,127.0.71.85:39963,127.0.71.86:37273,127.0.71.87:41751,127.0.71.88:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:52:39.132787 12974 catalog_manager.cc:1308] Loaded cluster ID: 3ed8461d403447e3800fefe76334ccef
I20260929 01:52:39.132906 12974 catalog_manager.cc:1601] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d: loading cluster ID for follower catalog manager: success
I20260929 01:52:39.137399 12974 catalog_manager.cc:1623] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d: acquiring CA information for follower catalog manager: success
I20260929 01:52:39.140486 12974 catalog_manager.cc:1651] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:52:39.166772 13129 heartbeater.cc:344] Connected to a master server at 127.0.71.124:40873
I20260929 01:52:39.167202 13129 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:39.167667 13131 heartbeater.cc:344] Connected to a master server at 127.0.71.125:44797
I20260929 01:52:39.167822 13131 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:39.168080 13130 heartbeater.cc:344] Connected to a master server at 127.0.71.126:42257
I20260929 01:52:39.168231 13130 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:39.168776 13129 heartbeater.cc:507] Master 127.0.71.124:40873 requested a full tablet report, sending...
I20260929 01:52:39.168954 13130 heartbeater.cc:507] Master 127.0.71.126:42257 requested a full tablet report, sending...
I20260929 01:52:39.169230 13131 heartbeater.cc:507] Master 127.0.71.125:44797 requested a full tablet report, sending...
I20260929 01:52:39.171437 12836 ts_manager.cc:194] Registered new tserver with Master: f796548a8f984835bc232e10b6ad1f76 (127.0.71.65:44525)
I20260929 01:52:39.171511 12908 ts_manager.cc:194] Registered new tserver with Master: f796548a8f984835bc232e10b6ad1f76 (127.0.71.65:44525)
I20260929 01:52:39.174074 12766 ts_manager.cc:194] Registered new tserver with Master: f796548a8f984835bc232e10b6ad1f76 (127.0.71.65:44525)
I20260929 01:52:39.176261 12766 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:35195
I20260929 01:52:39.481299 12996 catalog_manager.cc:1308] Loaded cluster ID: 3ed8461d403447e3800fefe76334ccef
I20260929 01:52:39.481429 12996 catalog_manager.cc:1601] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a: loading cluster ID for follower catalog manager: success
I20260929 01:52:39.485481 12996 catalog_manager.cc:1623] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a: acquiring CA information for follower catalog manager: success
I20260929 01:52:39.487983 12996 catalog_manager.cc:1651] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 01:52:39.492668 13139 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:39.492931 13139 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:39.493008 13139 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:39.501219 13139 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:39.501364 13139 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.66
I20260929 01:52:39.511104 13139 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35469,127.0.71.85:39963,127.0.71.86:37273,127.0.71.87:41751,127.0.71.88:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:39.512794 13139 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:39.514618 13139 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:39.524690 13144 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:39.524663 13145 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:39.525070 13147 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:39.526242 13139 server_base.cc:1061] running on GCE node
I20260929 01:52:39.526875 13139 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:39.528146 13139 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:39.529572 13139 hybrid_clock.cc:648] HybridClock initialized: now 1790646759529326 us; error 144 us; skew 500 ppm
I20260929 01:52:39.529920 13139 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:39.532243 13139 webserver.cc:533] Webserver started at http://127.0.71.66:45975/ using document root <none> and password file <none>
I20260929 01:52:39.533111 13139 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:39.533257 13139 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:39.533653 13139 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:52:39.536193 13139 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/data/instance:
uuid: "289825aadbca44a281341ab141d3d995"
format_stamp: "Formatted at 2026-09-29 01:52:39 on dist-test-slave-vz23"
I20260929 01:52:39.537122 13139 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/wal/instance:
uuid: "289825aadbca44a281341ab141d3d995"
format_stamp: "Formatted at 2026-09-29 01:52:39 on dist-test-slave-vz23"
I20260929 01:52:39.543072 13139 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 01:52:39.546958 13153 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:39.548906 13139 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 01:52:39.549136 13139 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/wal
uuid: "289825aadbca44a281341ab141d3d995"
format_stamp: "Formatted at 2026-09-29 01:52:39 on dist-test-slave-vz23"
I20260929 01:52:39.549393 13139 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:39.585819 13139 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:39.586777 13139 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:39.587037 13139 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:39.588014 13139 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:52:39.590055 13139 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:52:39.590160 13139 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:39.590281 13139 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:52:39.590385 13139 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:39.629068 13139 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.66:46753
I20260929 01:52:39.629127 13265 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.66:46753 every 8 connection(s)
I20260929 01:52:39.630713 13139 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
I20260929 01:52:39.640030 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 13139
I20260929 01:52:39.640266 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/wal/instance
I20260929 01:52:39.646428 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.67:0
--local_ip_for_outbound_sockets=127.0.71.67
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--builtin_ntp_servers=127.0.71.84:35469,127.0.71.85:39963,127.0.71.86:37273,127.0.71.87:41751,127.0.71.88:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:52:39.653256 13268 heartbeater.cc:344] Connected to a master server at 127.0.71.125:44797
I20260929 01:52:39.653278 13267 heartbeater.cc:344] Connected to a master server at 127.0.71.126:42257
I20260929 01:52:39.653705 13267 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:39.653784 13266 heartbeater.cc:344] Connected to a master server at 127.0.71.124:40873
I20260929 01:52:39.653909 13266 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:39.653713 13268 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:39.654894 13267 heartbeater.cc:507] Master 127.0.71.126:42257 requested a full tablet report, sending...
I20260929 01:52:39.655009 13268 heartbeater.cc:507] Master 127.0.71.125:44797 requested a full tablet report, sending...
I20260929 01:52:39.656098 13266 heartbeater.cc:507] Master 127.0.71.124:40873 requested a full tablet report, sending...
I20260929 01:52:39.656734 12766 ts_manager.cc:194] Registered new tserver with Master: 289825aadbca44a281341ab141d3d995 (127.0.71.66:46753)
I20260929 01:52:39.657150 12908 ts_manager.cc:194] Registered new tserver with Master: 289825aadbca44a281341ab141d3d995 (127.0.71.66:46753)
I20260929 01:52:39.657783 12766 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.66:40665
I20260929 01:52:39.658625 12836 ts_manager.cc:194] Registered new tserver with Master: 289825aadbca44a281341ab141d3d995 (127.0.71.66:46753)
W20260929 01:52:39.982726 13273 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:39.983055 13273 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:39.983225 13273 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:39.991846 13273 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:39.992063 13273 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.67
I20260929 01:52:40.002414 13273 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35469,127.0.71.85:39963,127.0.71.86:37273,127.0.71.87:41751,127.0.71.88:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:40.004040 13273 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:40.005882 13273 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:40.016081 13278 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:40.016189 13279 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:40.016103 13281 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:40.017967 13273 server_base.cc:1061] running on GCE node
I20260929 01:52:40.018699 13273 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:40.019996 13273 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:40.021224 13273 hybrid_clock.cc:648] HybridClock initialized: now 1790646760021104 us; error 89 us; skew 500 ppm
I20260929 01:52:40.021569 13273 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:40.023677 13273 webserver.cc:533] Webserver started at http://127.0.71.67:40353/ using document root <none> and password file <none>
I20260929 01:52:40.024428 13273 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:40.024617 13273 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:40.025010 13273 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:52:40.027429 13273 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/data/instance:
uuid: "f3b2a9413c9e40d59ae47fcc88e6abf3"
format_stamp: "Formatted at 2026-09-29 01:52:40 on dist-test-slave-vz23"
I20260929 01:52:40.028189 13273 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/wal/instance:
uuid: "f3b2a9413c9e40d59ae47fcc88e6abf3"
format_stamp: "Formatted at 2026-09-29 01:52:40 on dist-test-slave-vz23"
I20260929 01:52:40.033798 13273 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 01:52:40.037360 13287 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:40.038887 13273 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:52:40.039083 13273 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/wal
uuid: "f3b2a9413c9e40d59ae47fcc88e6abf3"
format_stamp: "Formatted at 2026-09-29 01:52:40 on dist-test-slave-vz23"
I20260929 01:52:40.039314 13273 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:40.062062 13273 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:40.062918 13273 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:40.063202 13273 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:40.064297 13273 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:52:40.066082 13273 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:52:40.066206 13273 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:40.066345 13273 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:52:40.066437 13273 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:40.101830 13273 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.67:35585
I20260929 01:52:40.101922 13399 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.67:35585 every 8 connection(s)
I20260929 01:52:40.103274 13273 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
I20260929 01:52:40.110687 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 13273
I20260929 01:52:40.110893 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/wal/instance
I20260929 01:52:40.125936 13400 heartbeater.cc:344] Connected to a master server at 127.0.71.124:40873
I20260929 01:52:40.125965 13402 heartbeater.cc:344] Connected to a master server at 127.0.71.125:44797
I20260929 01:52:40.126031 13401 heartbeater.cc:344] Connected to a master server at 127.0.71.126:42257
I20260929 01:52:40.126361 13401 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:40.126364 13400 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:40.126367 13402 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:40.127358 13400 heartbeater.cc:507] Master 127.0.71.124:40873 requested a full tablet report, sending...
I20260929 01:52:40.127378 13401 heartbeater.cc:507] Master 127.0.71.126:42257 requested a full tablet report, sending...
I20260929 01:52:40.128407 13402 heartbeater.cc:507] Master 127.0.71.125:44797 requested a full tablet report, sending...
I20260929 01:52:40.129106 12908 ts_manager.cc:194] Registered new tserver with Master: f3b2a9413c9e40d59ae47fcc88e6abf3 (127.0.71.67:35585)
I20260929 01:52:40.129076 12766 ts_manager.cc:194] Registered new tserver with Master: f3b2a9413c9e40d59ae47fcc88e6abf3 (127.0.71.67:35585)
I20260929 01:52:40.129590 12836 ts_manager.cc:194] Registered new tserver with Master: f3b2a9413c9e40d59ae47fcc88e6abf3 (127.0.71.67:35585)
I20260929 01:52:40.130182 12766 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.67:43065
I20260929 01:52:40.137624 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:52:40.145282 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:52:40.145495 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 12735
I20260929 01:52:40.146492 12796 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:40.179893 13130 heartbeater.cc:499] Master 127.0.71.126:42257 was elected leader, sending a full tablet report...
I20260929 01:52:40.259527 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 12735
W20260929 01:52:40.280777 12821 connection.cc:573] server connection from 127.0.0.1:33934 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:52:40.281841 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:42257
--webserver_interface=127.0.71.126
--webserver_port=45621
--builtin_ntp_servers=127.0.71.84:35469,127.0.71.85:39963,127.0.71.86:37273,127.0.71.87:41751,127.0.71.88:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873 with env {}
W20260929 01:52:40.617666 13408 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:40.617965 13408 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:40.618033 13408 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:40.626609 13408 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:52:40.626703 13408 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:40.626761 13408 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:52:40.626801 13408 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:52:40.637269 13408 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35469,127.0.71.85:39963,127.0.71.86:37273,127.0.71.87:41751,127.0.71.88:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:42257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=45621
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:40.638742 13408 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:40.640409 13408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:40.648772 13413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:40.649045 13414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:40.649509 13416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:40.650393 13408 server_base.cc:1061] running on GCE node
I20260929 01:52:40.651154 13408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:40.652809 13408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:40.654008 13408 hybrid_clock.cc:648] HybridClock initialized: now 1790646760653860 us; error 116 us; skew 500 ppm
I20260929 01:52:40.654379 13408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:40.656684 13408 webserver.cc:533] Webserver started at http://127.0.71.126:45621/ using document root <none> and password file <none>
I20260929 01:52:40.657507 13408 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:40.657660 13408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:40.660557 13267 heartbeater.cc:499] Master 127.0.71.126:42257 was elected leader, sending a full tablet report...
W20260929 01:52:40.661926 13267 heartbeater.cc:646] Failed to heartbeat to 127.0.71.126:42257 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.71.126:42257: connect: Connection refused (error 111)
I20260929 01:52:40.664055 13408 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.003s
I20260929 01:52:40.667089 13423 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:40.668591 13408 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 01:52:40.668774 13408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "44f4988489a442e989d3e74837e674ad"
format_stamp: "Formatted at 2026-09-29 01:52:37 on dist-test-slave-vz23"
I20260929 01:52:40.669533 13408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:40.700657 13408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:40.701629 13408 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:40.701967 13408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:40.724009 13474 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:42257 every 8 connection(s)
I20260929 01:52:40.724005 13408 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:42257
I20260929 01:52:40.725905 13408 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:52:40.732532 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 13408
I20260929 01:52:40.732880 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 13001
I20260929 01:52:40.733371 13475 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:52:40.737915 13475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: Bootstrap starting.
I20260929 01:52:40.743127 13124 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:40.757062 13475 log.cc:826] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: Log is configured to *not* fsync() on all Append() calls
I20260929 01:52:40.765892 13475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:52:40.766455 13475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: Bootstrap complete.
I20260929 01:52:40.773240 13475 raft_consensus.cc:359] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:40.773931 13475 raft_consensus.cc:740] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f4988489a442e989d3e74837e674ad, State: Initialized, Role: FOLLOWER
I20260929 01:52:40.774688 13475 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:40.775640 13481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:40.775921 13481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:40.776782 13475 sys_catalog.cc:565] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:52:40.787278 13492 catalog_manager.cc:1308] Loaded cluster ID: 3ed8461d403447e3800fefe76334ccef
I20260929 01:52:40.787386 13492 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: loading cluster ID for follower catalog manager: success
I20260929 01:52:40.792685 13492 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: acquiring CA information for follower catalog manager: success
I20260929 01:52:40.795336 13492 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:52:40.821841 13408 master_runner.cc:264] No leader master found from master 44f4988489a442e989d3e74837e674ad
I20260929 01:52:40.822022 13408 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:52:40.866286 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 13001
I20260929 01:52:40.884234 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:44525
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--webserver_port=34699
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:35469,127.0.71.85:39963,127.0.71.86:37273,127.0.71.87:41751,127.0.71.88:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:52:41.132586 13401 heartbeater.cc:499] Master 127.0.71.126:42257 was elected leader, sending a full tablet report...
I20260929 01:52:41.144137 13440 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f3b2a9413c9e40d59ae47fcc88e6abf3" instance_seqno: 1790646760093709) as {username='slave'} at 127.0.71.67:44331; Asking this server to re-register.
I20260929 01:52:41.145591 13401 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:41.147684 13440 ts_manager.cc:194] Registered new tserver with Master: f3b2a9413c9e40d59ae47fcc88e6abf3 (127.0.71.67:35585)
W20260929 01:52:41.266772 13498 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:41.267113 13498 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:41.267196 13498 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:41.276233 13498 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:41.276415 13498 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:52:41.286999 13498 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35469,127.0.71.85:39963,127.0.71.86:37273,127.0.71.87:41751,127.0.71.88:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:44525
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=34699
--tserver_master_addrs=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:41.288687 13498 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:41.290509 13498 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:41.300956 13504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:41.301010 13507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:41.300987 13505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:41.301805 13498 server_base.cc:1061] running on GCE node
I20260929 01:52:41.302399 13498 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:41.303577 13498 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:41.304867 13498 hybrid_clock.cc:648] HybridClock initialized: now 1790646761304727 us; error 104 us; skew 500 ppm
I20260929 01:52:41.305234 13498 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:41.307436 13498 webserver.cc:533] Webserver started at http://127.0.71.65:34699/ using document root <none> and password file <none>
I20260929 01:52:41.308226 13498 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:41.308363 13498 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:41.313843 13498 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 01:52:41.316761 13513 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:41.318035 13498 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:52:41.318223 13498 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "f796548a8f984835bc232e10b6ad1f76"
format_stamp: "Formatted at 2026-09-29 01:52:39 on dist-test-slave-vz23"
I20260929 01:52:41.319116 13498 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:41.348897 13498 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:41.349942 13498 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:41.350319 13498 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:41.351390 13498 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:52:41.353266 13498 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:52:41.353386 13498 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:41.353565 13498 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:52:41.353648 13498 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:41.391124 13498 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:44525
I20260929 01:52:41.391183 13625 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:44525 every 8 connection(s)
I20260929 01:52:41.392843 13498 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:52:41.400552 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 13498
I20260929 01:52:41.409816 13626 heartbeater.cc:344] Connected to a master server at 127.0.71.124:40873
I20260929 01:52:41.410343 13626 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:41.410925 13627 heartbeater.cc:344] Connected to a master server at 127.0.71.126:42257
I20260929 01:52:41.411098 13627 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:41.411783 13627 heartbeater.cc:507] Master 127.0.71.126:42257 requested a full tablet report, sending...
I20260929 01:52:41.411819 13626 heartbeater.cc:507] Master 127.0.71.124:40873 requested a full tablet report, sending...
I20260929 01:52:41.412675 13628 heartbeater.cc:344] Connected to a master server at 127.0.71.125:44797
I20260929 01:52:41.412823 13628 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:41.413264 13628 heartbeater.cc:507] Master 127.0.71.125:44797 requested a full tablet report, sending...
I20260929 01:52:41.413836 12908 ts_manager.cc:194] Re-registered known tserver with Master: f796548a8f984835bc232e10b6ad1f76 (127.0.71.65:44525)
I20260929 01:52:41.414337 13440 ts_manager.cc:194] Registered new tserver with Master: f796548a8f984835bc232e10b6ad1f76 (127.0.71.65:44525)
I20260929 01:52:41.414728 12836 ts_manager.cc:194] Re-registered known tserver with Master: f796548a8f984835bc232e10b6ad1f76 (127.0.71.65:44525)
I20260929 01:52:41.446851 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 13498
I20260929 01:52:41.465890 13621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:41.550369 13639 raft_consensus.cc:493] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44f4988489a442e989d3e74837e674ad)
I20260929 01:52:41.550668 13639 raft_consensus.cc:515] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:41.552191 13639 leader_election.cc:290] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 44f4988489a442e989d3e74837e674ad (127.0.71.126:42257), d7efd97cc2a740c0a88ee4a714d6748a (127.0.71.125:44797)
I20260929 01:52:41.553053 12847 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae67ac031b4c4b84a8b717d053cb563d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" is_pre_election: true
I20260929 01:52:41.558696 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 13498
I20260929 01:52:41.563694 13450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae67ac031b4c4b84a8b717d053cb563d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "44f4988489a442e989d3e74837e674ad" is_pre_election: true
I20260929 01:52:41.564241 13450 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae67ac031b4c4b84a8b717d053cb563d in term 1.
I20260929 01:52:41.564869 12895 leader_election.cc:304] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44f4988489a442e989d3e74837e674ad, ae67ac031b4c4b84a8b717d053cb563d; no voters: d7efd97cc2a740c0a88ee4a714d6748a
I20260929 01:52:41.565335 13639 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 01:52:41.565483 13639 raft_consensus.cc:493] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 1 FOLLOWER]: Starting leader election (detected failure of leader 44f4988489a442e989d3e74837e674ad)
I20260929 01:52:41.565603 13639 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:52:41.569866 13639 raft_consensus.cc:515] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:41.570765 13639 leader_election.cc:290] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [CANDIDATE]: Term 2 election: Requested vote from peers 44f4988489a442e989d3e74837e674ad (127.0.71.126:42257), d7efd97cc2a740c0a88ee4a714d6748a (127.0.71.125:44797)
I20260929 01:52:41.571194 13450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae67ac031b4c4b84a8b717d053cb563d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "44f4988489a442e989d3e74837e674ad"
I20260929 01:52:41.571339 12847 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae67ac031b4c4b84a8b717d053cb563d" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7efd97cc2a740c0a88ee4a714d6748a"
I20260929 01:52:41.571429 13450 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:52:41.575515 13450 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae67ac031b4c4b84a8b717d053cb563d in term 2.
I20260929 01:52:41.575994 12895 leader_election.cc:304] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44f4988489a442e989d3e74837e674ad, ae67ac031b4c4b84a8b717d053cb563d; no voters: d7efd97cc2a740c0a88ee4a714d6748a
I20260929 01:52:41.576365 13639 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:52:41.576849 13639 raft_consensus.cc:697] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 2 LEADER]: Becoming Leader. State: Replica: ae67ac031b4c4b84a8b717d053cb563d, State: Running, Role: LEADER
I20260929 01:52:41.577450 13639 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:41.579710 13641 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae67ac031b4c4b84a8b717d053cb563d. Latest consensus state: current_term: 2 leader_uuid: "ae67ac031b4c4b84a8b717d053cb563d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:41.580093 13641 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [sys.catalog]: This master's current role is: LEADER
I20260929 01:52:41.581434 13643 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:52:41.581636 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:44525
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--webserver_port=34699
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:35469,127.0.71.85:39963,127.0.71.86:37273,127.0.71.87:41751,127.0.71.88:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {}
I20260929 01:52:41.584273 13643 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:52:41.585665 13643 catalog_manager.cc:1308] Loaded cluster ID: 3ed8461d403447e3800fefe76334ccef
I20260929 01:52:41.585760 13643 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:52:41.587491 13643 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:52:41.588766 13643 catalog_manager.cc:6230] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d: Loaded TSK: 0
I20260929 01:52:41.589759 13643 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:52:41.669381 13267 heartbeater.cc:344] Connected to a master server at 127.0.71.126:42257
I20260929 01:52:41.669622 13267 heartbeater.cc:499] Master 127.0.71.126:42257 was elected leader, sending a full tablet report...
I20260929 01:52:41.670733 13440 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "289825aadbca44a281341ab141d3d995" instance_seqno: 1790646759620203) as {username='slave'} at 127.0.71.66:42161; Asking this server to re-register.
I20260929 01:52:41.671383 13267 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:41.672659 13440 ts_manager.cc:194] Registered new tserver with Master: 289825aadbca44a281341ab141d3d995 (127.0.71.66:46753)
I20260929 01:52:41.761189 13646 raft_consensus.cc:493] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 44f4988489a442e989d3e74837e674ad)
I20260929 01:52:41.761471 13646 raft_consensus.cc:515] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:41.763034 13646 leader_election.cc:290] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 44f4988489a442e989d3e74837e674ad (127.0.71.126:42257), ae67ac031b4c4b84a8b717d053cb563d (127.0.71.124:40873)
I20260929 01:52:41.763959 12917 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ae67ac031b4c4b84a8b717d053cb563d" is_pre_election: true
I20260929 01:52:41.770848 13450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "44f4988489a442e989d3e74837e674ad" is_pre_election: true
I20260929 01:52:41.771200 13450 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7efd97cc2a740c0a88ee4a714d6748a in current term 2: Already voted for candidate ae67ac031b4c4b84a8b717d053cb563d in this term.
I20260929 01:52:41.771878 12824 leader_election.cc:304] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7efd97cc2a740c0a88ee4a714d6748a; no voters: 44f4988489a442e989d3e74837e674ad, ae67ac031b4c4b84a8b717d053cb563d
I20260929 01:52:41.772279 13646 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:52:41.775936 13646 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
W20260929 01:52:41.923060 13644 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:41.923353 13644 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:41.923501 13644 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:52:41.923592 13644 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:41.931854 13644 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:41.932103 13644 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:52:41.942219 13644 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35469,127.0.71.85:39963,127.0.71.86:37273,127.0.71.87:41751,127.0.71.88:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:44525
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=34699
--tserver_master_addrs=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:41.944003 13644 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:41.945845 13644 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:41.955216 13652 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:41.955451 13653 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:41.956115 13655 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:41.957150 13644 server_base.cc:1061] running on GCE node
I20260929 01:52:41.957719 13644 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:41.958837 13644 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:41.960111 13644 hybrid_clock.cc:648] HybridClock initialized: now 1790646761959942 us; error 126 us; skew 500 ppm
I20260929 01:52:41.960403 13644 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:41.962515 13644 webserver.cc:533] Webserver started at http://127.0.71.65:34699/ using document root <none> and password file <none>
I20260929 01:52:41.963224 13644 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:41.963375 13644 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:41.968535 13644 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:52:41.971269 13661 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:41.972721 13644 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:52:41.972906 13644 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "f796548a8f984835bc232e10b6ad1f76"
format_stamp: "Formatted at 2026-09-29 01:52:39 on dist-test-slave-vz23"
I20260929 01:52:41.973631 13644 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:41.997915 13644 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:41.998953 13644 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:41.999275 13644 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:42.000315 13644 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:52:42.002125 13644 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:52:42.002262 13644 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:42.002424 13644 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:52:42.002537 13644 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:52:42.002935 13644 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:52:42.003072 13644 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:52:42.003165 13644 fault_injection.cc:59] injecting fault for kudu (pid 13644): FLAGS_fault_before_start (process will exit)
I20260929 01:52:42.009482 13450 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 2 FOLLOWER]: Refusing update from remote peer ae67ac031b4c4b84a8b717d053cb563d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 01:52:42.010402 13641 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:52:42.014241 13669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae67ac031b4c4b84a8b717d053cb563d. Latest consensus state: current_term: 2 leader_uuid: "ae67ac031b4c4b84a8b717d053cb563d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:42.014595 13669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:42.015664 13639 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ae67ac031b4c4b84a8b717d053cb563d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:42.015949 13639 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [sys.catalog]: This master's current role is: LEADER
I20260929 01:52:42.016891 12847 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [term 2 FOLLOWER]: Refusing update from remote peer ae67ac031b4c4b84a8b717d053cb563d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 01:52:42.017086 13669 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ae67ac031b4c4b84a8b717d053cb563d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:42.017298 13669 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:42.017642 13639 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:52:42.020273 13676 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ae67ac031b4c4b84a8b717d053cb563d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:42.020352 13646 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae67ac031b4c4b84a8b717d053cb563d. Latest consensus state: current_term: 2 leader_uuid: "ae67ac031b4c4b84a8b717d053cb563d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:42.020612 13676 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:42.020635 13646 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:42.025327 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:44525
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--webserver_port=34699
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:35469,127.0.71.85:39963,127.0.71.86:37273,127.0.71.87:41751,127.0.71.88:43105
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260929 01:52:42.375813 13677 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:42.376164 13677 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:42.376302 13677 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:42.384789 13677 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:42.385005 13677 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:52:42.395025 13677 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35469,127.0.71.85:39963,127.0.71.86:37273,127.0.71.87:41751,127.0.71.88:43105
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:44525
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=34699
--tserver_master_addrs=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:42.396790 13677 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:42.398670 13677 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:42.408522 13682 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:42.408485 13683 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:42.409554 13685 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:42.410043 13677 server_base.cc:1061] running on GCE node
I20260929 01:52:42.410686 13677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:42.411877 13677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:42.413259 13677 hybrid_clock.cc:648] HybridClock initialized: now 1790646762413087 us; error 129 us; skew 500 ppm
I20260929 01:52:42.413697 13677 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:42.416106 13677 webserver.cc:533] Webserver started at http://127.0.71.65:34699/ using document root <none> and password file <none>
I20260929 01:52:42.416996 13677 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:42.417132 13677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:42.422978 13677 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:52:42.425781 13691 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:42.427140 13677 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:52:42.427310 13677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "f796548a8f984835bc232e10b6ad1f76"
format_stamp: "Formatted at 2026-09-29 01:52:39 on dist-test-slave-vz23"
I20260929 01:52:42.428059 13677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:42.442239 13677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:42.443037 13677 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:42.443327 13677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:42.444351 13677 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:52:42.446033 13677 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:52:42.446166 13677 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 01:52:42.446313 13677 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:52:42.446375 13677 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:42.466995 13672 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ae67ac031b4c4b84a8b717d053cb563d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:42.467322 13672 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [sys.catalog]: This master's current role is: LEADER
I20260929 01:52:42.483697 13677 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:44525
I20260929 01:52:42.483729 13803 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:44525 every 8 connection(s)
I20260929 01:52:42.485363 13677 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:52:42.486869 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 13677
I20260929 01:52:42.487221 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 13677
I20260929 01:52:42.518831 13805 heartbeater.cc:344] Connected to a master server at 127.0.71.126:42257
I20260929 01:52:42.519294 13805 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:42.520035 13806 heartbeater.cc:344] Connected to a master server at 127.0.71.125:44797
I20260929 01:52:42.520195 13806 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:42.520746 13805 heartbeater.cc:507] Master 127.0.71.126:42257 requested a full tablet report, sending...
I20260929 01:52:42.520797 13799 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:42.523185 13440 ts_manager.cc:194] Re-registered known tserver with Master: f796548a8f984835bc232e10b6ad1f76 (127.0.71.65:44525)
I20260929 01:52:42.610419 13804 heartbeater.cc:344] Connected to a master server at 127.0.71.124:40873
I20260929 01:52:42.610440 13806 heartbeater.cc:507] Master 127.0.71.125:44797 requested a full tablet report, sending...
I20260929 01:52:42.610664 13804 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:42.611364 13804 heartbeater.cc:507] Master 127.0.71.124:40873 requested a full tablet report, sending...
I20260929 01:52:42.611991 12836 ts_manager.cc:194] Re-registered known tserver with Master: f796548a8f984835bc232e10b6ad1f76 (127.0.71.65:44525)
I20260929 01:52:42.612485 12908 ts_manager.cc:194] Re-registered known tserver with Master: f796548a8f984835bc232e10b6ad1f76 (127.0.71.65:44525)
I20260929 01:52:42.613665 12908 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:50451
I20260929 01:52:42.614794 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 13677
I20260929 01:52:42.631575 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 13139
I20260929 01:52:42.639879 13261 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:42.728215 13266 heartbeater.cc:499] Master 127.0.71.124:40873 was elected leader, sending a full tablet report...
I20260929 01:52:42.730583 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 13139
W20260929 01:52:42.747263 13425 connection.cc:573] server connection from 127.0.71.66:42161 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:52:42.747690 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 13273
I20260929 01:52:42.755867 13395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:42.843608 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 13273
I20260929 01:52:42.859963 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 13408
I20260929 01:52:42.866079 13470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:42.977468 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 13408
I20260929 01:52:42.997262 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 12803
I20260929 01:52:42.998451 12867 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:43.116266 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 12803
I20260929 01:52:43.135900 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 12874
I20260929 01:52:43.137079 12938 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:43.249764 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 12874
Shutting down hive metastore.
2026-09-29T01:52:43Z chronyd exiting
2026-09-29T01:52:43Z chronyd exiting
2026-09-29T01:52:43Z chronyd exiting
2026-09-29T01:52:43Z chronyd exiting
2026-09-29T01:52:43Z chronyd exiting
2026-09-29T01:52:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:52:43Z Disabled control of system clock
2026-09-29T01:52:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:52:43Z Disabled control of system clock
2026-09-29T01:52:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:52:43Z Disabled control of system clock
2026-09-29T01:52:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:52:43Z Disabled control of system clock
2026-09-29T01:52:43Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:52:43Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/security.properties
2026-09-29 01:52:44: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260929 01:52:53.017444 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:42257
--webserver_interface=127.0.71.126
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:42823,127.0.71.85:37899,127.0.71.86:36353,127.0.71.87:45495,127.0.71.88:37305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873 with env {}
W20260929 01:52:53.347421 14015 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:53.347787 14015 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:53.347914 14015 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:53.356248 14015 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:52:53.356374 14015 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:53.356515 14015 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:52:53.356609 14015 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:52:53.367002 14015 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:42823,127.0.71.85:37899,127.0.71.86:36353,127.0.71.87:45495,127.0.71.88:37305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:42257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:53.368695 14015 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:53.370474 14015 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:53.379047 14021 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:53.379345 14015 server_base.cc:1061] running on GCE node
W20260929 01:52:53.379047 14020 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:53.379089 14023 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:53.380616 14015 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:53.382412 14015 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:53.383669 14015 hybrid_clock.cc:648] HybridClock initialized: now 1790646773383537 us; error 99 us; skew 500 ppm
I20260929 01:52:53.384043 14015 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:53.386474 14015 webserver.cc:533] Webserver started at http://127.0.71.126:38495/ using document root <none> and password file <none>
I20260929 01:52:53.387214 14015 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:53.387356 14015 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:53.392911 14015 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:52:53.397461 14029 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:53.398874 14015 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:52:53.399068 14015 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "44f4988489a442e989d3e74837e674ad"
format_stamp: "Formatted at 2026-09-29 01:52:37 on dist-test-slave-vz23"
I20260929 01:52:53.399751 14015 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:53.427901 14015 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:53.428877 14015 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:53.429203 14015 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:53.450042 14015 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:42257
I20260929 01:52:53.450083 14080 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:42257 every 8 connection(s)
I20260929 01:52:53.451737 14015 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:52:53.457973 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 14015
I20260929 01:52:53.458173 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:52:53.458738 14081 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:52:53.459232 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:44797
--webserver_interface=127.0.71.125
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:42823,127.0.71.85:37899,127.0.71.86:36353,127.0.71.87:45495,127.0.71.88:37305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873 with env {}
I20260929 01:52:53.462214 14081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: Bootstrap starting.
I20260929 01:52:53.478449 14081 log.cc:826] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: Log is configured to *not* fsync() on all Append() calls
I20260929 01:52:53.487273 14081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:52:53.487839 14081 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: Bootstrap complete.
I20260929 01:52:53.494516 14081 raft_consensus.cc:359] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:53.495154 14081 raft_consensus.cc:740] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f4988489a442e989d3e74837e674ad, State: Initialized, Role: FOLLOWER
I20260929 01:52:53.495932 14081 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:53.496836 14085 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:53.497138 14085 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:53.497812 14081 sys_catalog.cc:565] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:52:53.508241 14096 catalog_manager.cc:1308] Loaded cluster ID: 3ed8461d403447e3800fefe76334ccef
I20260929 01:52:53.508463 14096 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: loading cluster ID for follower catalog manager: success
I20260929 01:52:53.512830 14096 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: acquiring CA information for follower catalog manager: success
I20260929 01:52:53.515547 14096 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:52:53.529203 14015 master_runner.cc:187] Error getting master registration for 127.0.71.125:44797: Network error: Client connection negotiation failed: client connection to 127.0.71.125:44797: connect: Connection refused (error 111),
I20260929 01:52:53.534961 14015 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:52:53.803872 14083 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:53.804168 14083 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:53.804311 14083 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:53.812908 14083 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:52:53.813004 14083 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:53.813098 14083 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:52:53.813171 14083 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:52:53.823598 14083 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:42823,127.0.71.85:37899,127.0.71.86:36353,127.0.71.87:45495,127.0.71.88:37305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:44797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:53.825399 14083 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:53.827234 14083 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:53.835384 14104 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:53.835961 14105 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:53.837402 14107 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:53.842279 14083 server_base.cc:1061] running on GCE node
I20260929 01:52:53.843201 14083 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:53.845129 14083 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:53.846565 14083 hybrid_clock.cc:648] HybridClock initialized: now 1790646773846430 us; error 101 us; skew 500 ppm
I20260929 01:52:53.847002 14083 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:53.849423 14083 webserver.cc:533] Webserver started at http://127.0.71.125:42681/ using document root <none> and password file <none>
I20260929 01:52:53.850175 14083 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:53.850325 14083 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:53.856297 14083 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:52:53.859546 14113 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:53.861100 14083 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:52:53.861302 14083 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "d7efd97cc2a740c0a88ee4a714d6748a"
format_stamp: "Formatted at 2026-09-29 01:52:37 on dist-test-slave-vz23"
I20260929 01:52:53.862061 14083 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:53.877085 14083 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:53.877990 14083 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:53.878301 14083 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:53.899219 14083 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:44797
I20260929 01:52:53.899230 14164 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:44797 every 8 connection(s)
I20260929 01:52:53.900950 14083 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:52:53.908833 14165 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:52:53.909525 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 14083
I20260929 01:52:53.909761 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:52:53.910668 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:40873
--webserver_interface=127.0.71.124
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:42823,127.0.71.85:37899,127.0.71.86:36353,127.0.71.87:45495,127.0.71.88:37305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873 with env {}
I20260929 01:52:53.912904 14165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a: Bootstrap starting.
I20260929 01:52:53.936864 14165 log.cc:826] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a: Log is configured to *not* fsync() on all Append() calls
I20260929 01:52:53.948402 14165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:52:53.949105 14165 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a: Bootstrap complete.
I20260929 01:52:53.959357 14165 raft_consensus.cc:359] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:53.960351 14165 raft_consensus.cc:740] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7efd97cc2a740c0a88ee4a714d6748a, State: Initialized, Role: FOLLOWER
I20260929 01:52:53.961267 14165 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:53.962123 14169 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:53.962399 14169 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:53.963577 14165 sys_catalog.cc:565] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:52:53.974150 14180 catalog_manager.cc:1308] Loaded cluster ID: 3ed8461d403447e3800fefe76334ccef
I20260929 01:52:53.974330 14180 catalog_manager.cc:1601] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a: loading cluster ID for follower catalog manager: success
I20260929 01:52:53.978410 14180 catalog_manager.cc:1623] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a: acquiring CA information for follower catalog manager: success
I20260929 01:52:53.981112 14180 catalog_manager.cc:1651] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:52:54.016868 14083 master_runner.cc:187] Error getting master registration for 127.0.71.124:40873: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40873: connect: Connection refused (error 111),
I20260929 01:52:54.023747 14083 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:52:54.286275 14167 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:54.286597 14167 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:54.286734 14167 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:54.295310 14167 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:52:54.295413 14167 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:54.295496 14167 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:52:54.295585 14167 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:52:54.306581 14167 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:42823,127.0.71.85:37899,127.0.71.86:36353,127.0.71.87:45495,127.0.71.88:37305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:40873
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:54.308197 14167 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:54.310137 14167 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:54.318961 14188 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:54.319613 14189 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:54.321240 14191 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:54.322007 14167 server_base.cc:1061] running on GCE node
I20260929 01:52:54.322834 14167 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:54.324594 14167 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:54.325847 14167 hybrid_clock.cc:648] HybridClock initialized: now 1790646774325691 us; error 132 us; skew 500 ppm
I20260929 01:52:54.326213 14167 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:54.328863 14167 webserver.cc:533] Webserver started at http://127.0.71.124:39351/ using document root <none> and password file <none>
I20260929 01:52:54.329617 14167 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:54.329766 14167 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:54.335306 14167 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 01:52:54.338387 14197 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:54.339846 14167 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:52:54.340030 14167 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "ae67ac031b4c4b84a8b717d053cb563d"
format_stamp: "Formatted at 2026-09-29 01:52:37 on dist-test-slave-vz23"
I20260929 01:52:54.340751 14167 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:54.358109 14167 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:54.359103 14167 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:54.359397 14167 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:54.381695 14248 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:40873 every 8 connection(s)
I20260929 01:52:54.381705 14167 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:40873
I20260929 01:52:54.383538 14167 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:52:54.386003 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 14167
I20260929 01:52:54.386181 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:52:54.393426 14249 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:52:54.398005 14249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d: Bootstrap starting.
I20260929 01:52:54.416345 14249 log.cc:826] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:52:54.426616 14249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:52:54.427374 14249 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d: Bootstrap complete.
I20260929 01:52:54.433804 14249 raft_consensus.cc:359] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:54.434530 14249 raft_consensus.cc:740] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae67ac031b4c4b84a8b717d053cb563d, State: Initialized, Role: FOLLOWER
I20260929 01:52:54.435222 14249 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:54.436061 14254 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:54.436359 14254 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:54.437091 14249 sys_catalog.cc:565] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:52:54.447796 14265 catalog_manager.cc:1308] Loaded cluster ID: 3ed8461d403447e3800fefe76334ccef
I20260929 01:52:54.447923 14265 catalog_manager.cc:1601] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d: loading cluster ID for follower catalog manager: success
I20260929 01:52:54.451738 14265 catalog_manager.cc:1623] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d: acquiring CA information for follower catalog manager: success
I20260929 01:52:54.455919 14265 catalog_manager.cc:1651] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:52:54.478366 14167 master_runner.cc:264] No leader master found from master ae67ac031b4c4b84a8b717d053cb563d
I20260929 01:52:54.478549 14167 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:52:54.792505 14268 raft_consensus.cc:493] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 2 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:52:54.792835 14268 raft_consensus.cc:515] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:54.794720 14268 leader_election.cc:290] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d7efd97cc2a740c0a88ee4a714d6748a (127.0.71.125:44797), ae67ac031b4c4b84a8b717d053cb563d (127.0.71.124:40873)
I20260929 01:52:54.803784 14140 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44f4988489a442e989d3e74837e674ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" is_pre_election: true
I20260929 01:52:54.804322 14140 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f4988489a442e989d3e74837e674ad in term 2.
I20260929 01:52:54.804708 14224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44f4988489a442e989d3e74837e674ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae67ac031b4c4b84a8b717d053cb563d" is_pre_election: true
I20260929 01:52:54.805124 14031 leader_election.cc:304] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f4988489a442e989d3e74837e674ad, d7efd97cc2a740c0a88ee4a714d6748a; no voters:
I20260929 01:52:54.805344 14224 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 44f4988489a442e989d3e74837e674ad in term 2.
I20260929 01:52:54.805583 14268 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 01:52:54.805737 14268 raft_consensus.cc:493] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 2 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:52:54.805868 14268 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:52:54.809706 14268 raft_consensus.cc:515] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:54.810477 14268 leader_election.cc:290] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [CANDIDATE]: Term 3 election: Requested vote from peers d7efd97cc2a740c0a88ee4a714d6748a (127.0.71.125:44797), ae67ac031b4c4b84a8b717d053cb563d (127.0.71.124:40873)
I20260929 01:52:54.811056 14140 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44f4988489a442e989d3e74837e674ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7efd97cc2a740c0a88ee4a714d6748a"
I20260929 01:52:54.811295 14140 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:52:54.812225 14224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "44f4988489a442e989d3e74837e674ad" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ae67ac031b4c4b84a8b717d053cb563d"
I20260929 01:52:54.812422 14224 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:52:54.816340 14224 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f4988489a442e989d3e74837e674ad in term 3.
I20260929 01:52:54.816717 14140 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 44f4988489a442e989d3e74837e674ad in term 3.
I20260929 01:52:54.816917 14030 leader_election.cc:304] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 44f4988489a442e989d3e74837e674ad, ae67ac031b4c4b84a8b717d053cb563d; no voters:
I20260929 01:52:54.817359 14268 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 3 FOLLOWER]: Leader election won for term 3
I20260929 01:52:54.817922 14268 raft_consensus.cc:697] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 3 LEADER]: Becoming Leader. State: Replica: 44f4988489a442e989d3e74837e674ad, State: Running, Role: LEADER
I20260929 01:52:54.818518 14268 consensus_queue.cc:237] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:54.821554 14271 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44f4988489a442e989d3e74837e674ad. Latest consensus state: current_term: 3 leader_uuid: "44f4988489a442e989d3e74837e674ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:54.821913 14271 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: This master's current role is: LEADER
I20260929 01:52:54.822593 14273 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:52:54.824710 14273 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:52:54.825505 14273 catalog_manager.cc:1308] Loaded cluster ID: 3ed8461d403447e3800fefe76334ccef
I20260929 01:52:54.825583 14273 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:52:54.826773 14273 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:52:54.827523 14273 catalog_manager.cc:6230] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: Loaded TSK: 0
I20260929 01:52:54.828397 14273 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:52:54.848732 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--builtin_ntp_servers=127.0.71.84:42823,127.0.71.85:37899,127.0.71.86:36353,127.0.71.87:45495,127.0.71.88:37305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260929 01:52:55.216193 14274 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:55.216537 14274 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:55.216704 14274 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:55.225238 14274 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:55.225399 14274 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:52:55.235661 14274 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:42823,127.0.71.85:37899,127.0.71.86:36353,127.0.71.87:45495,127.0.71.88:37305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:55.237541 14274 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:55.239506 14274 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:55.249558 14280 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:55.249518 14279 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:55.250537 14282 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:55.251098 14274 server_base.cc:1061] running on GCE node
I20260929 01:52:55.251724 14274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:55.252928 14274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:55.254202 14274 hybrid_clock.cc:648] HybridClock initialized: now 1790646775254052 us; error 119 us; skew 500 ppm
I20260929 01:52:55.254544 14274 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:55.256781 14274 webserver.cc:533] Webserver started at http://127.0.71.65:34417/ using document root <none> and password file <none>
I20260929 01:52:55.257546 14274 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:55.257637 14274 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:55.262914 14274 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:52:55.265758 14288 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:55.267159 14274 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:52:55.267318 14274 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "f796548a8f984835bc232e10b6ad1f76"
format_stamp: "Formatted at 2026-09-29 01:52:39 on dist-test-slave-vz23"
I20260929 01:52:55.268029 14274 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:55.294998 14274 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:55.295909 14274 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:55.296263 14274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:55.297358 14274 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:52:55.299096 14274 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:52:55.299177 14274 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:55.299372 14274 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:52:55.299440 14274 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:55.306767 14140 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [term 3 FOLLOWER]: Refusing update from remote peer 44f4988489a442e989d3e74837e674ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 01:52:55.307822 14271 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:52:55.310815 14313 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44f4988489a442e989d3e74837e674ad. Latest consensus state: current_term: 3 leader_uuid: "44f4988489a442e989d3e74837e674ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:55.311091 14313 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:55.312626 14224 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 3 FOLLOWER]: Refusing update from remote peer 44f4988489a442e989d3e74837e674ad: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 01:52:55.313620 14271 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:52:55.327194 14313 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "44f4988489a442e989d3e74837e674ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:55.327425 14313 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:55.329272 14325 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "44f4988489a442e989d3e74837e674ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:55.329609 14325 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:55.330964 14322 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 44f4988489a442e989d3e74837e674ad. Latest consensus state: current_term: 3 leader_uuid: "44f4988489a442e989d3e74837e674ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:55.331185 14322 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:55.332664 14327 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "44f4988489a442e989d3e74837e674ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:55.332901 14327 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: This master's current role is: LEADER
I20260929 01:52:55.332736 14268 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "44f4988489a442e989d3e74837e674ad" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:55.333065 14268 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: This master's current role is: LEADER
I20260929 01:52:55.361138 14274 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:39215
I20260929 01:52:55.361192 14410 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:39215 every 8 connection(s)
I20260929 01:52:55.362659 14274 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:52:55.367800 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 14274
I20260929 01:52:55.368057 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
I20260929 01:52:55.374788 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.66:0
--local_ip_for_outbound_sockets=127.0.71.66
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--builtin_ntp_servers=127.0.71.84:42823,127.0.71.85:37899,127.0.71.86:36353,127.0.71.87:45495,127.0.71.88:37305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:52:55.410131 14413 heartbeater.cc:344] Connected to a master server at 127.0.71.125:44797
I20260929 01:52:55.410197 14411 heartbeater.cc:344] Connected to a master server at 127.0.71.124:40873
I20260929 01:52:55.410530 14413 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:55.410540 14411 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:55.411844 14411 heartbeater.cc:507] Master 127.0.71.124:40873 requested a full tablet report, sending...
I20260929 01:52:55.411911 14413 heartbeater.cc:507] Master 127.0.71.125:44797 requested a full tablet report, sending...
I20260929 01:52:55.414557 14130 ts_manager.cc:194] Registered new tserver with Master: f796548a8f984835bc232e10b6ad1f76 (127.0.71.65:39215)
I20260929 01:52:55.414713 14214 ts_manager.cc:194] Registered new tserver with Master: f796548a8f984835bc232e10b6ad1f76 (127.0.71.65:39215)
I20260929 01:52:55.417017 14412 heartbeater.cc:344] Connected to a master server at 127.0.71.126:42257
I20260929 01:52:55.417169 14412 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:55.417605 14412 heartbeater.cc:507] Master 127.0.71.126:42257 requested a full tablet report, sending...
I20260929 01:52:55.419219 14046 ts_manager.cc:194] Registered new tserver with Master: f796548a8f984835bc232e10b6ad1f76 (127.0.71.65:39215)
I20260929 01:52:55.421481 14046 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:55887
W20260929 01:52:55.738880 14421 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:55.739214 14421 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:55.739297 14421 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:55.749686 14421 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:55.749866 14421 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.66
I20260929 01:52:55.760901 14421 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:42823,127.0.71.85:37899,127.0.71.86:36353,127.0.71.87:45495,127.0.71.88:37305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:55.762794 14421 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:55.764693 14421 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:55.774976 14426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:55.774740 14427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:55.776081 14429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:55.776211 14421 server_base.cc:1061] running on GCE node
I20260929 01:52:55.776866 14421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:55.778002 14421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:55.779213 14421 hybrid_clock.cc:648] HybridClock initialized: now 1790646775779076 us; error 108 us; skew 500 ppm
I20260929 01:52:55.779551 14421 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:55.781773 14421 webserver.cc:533] Webserver started at http://127.0.71.66:38603/ using document root <none> and password file <none>
I20260929 01:52:55.782526 14421 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:55.782670 14421 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:55.788398 14421 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 01:52:55.791342 14435 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:55.792827 14421 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:52:55.793025 14421 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/wal
uuid: "289825aadbca44a281341ab141d3d995"
format_stamp: "Formatted at 2026-09-29 01:52:39 on dist-test-slave-vz23"
I20260929 01:52:55.793740 14421 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:55.840242 14421 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:55.841192 14421 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:55.841492 14421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:55.842521 14421 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:52:55.844223 14421 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:52:55.844300 14421 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:55.844427 14421 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:52:55.844528 14421 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.001s
I20260929 01:52:55.880298 14421 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.66:36011
I20260929 01:52:55.880525 14547 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.66:36011 every 8 connection(s)
I20260929 01:52:55.881886 14421 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
I20260929 01:52:55.887816 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 14421
I20260929 01:52:55.888093 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-1/wal/instance
I20260929 01:52:55.893126 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.67:0
--local_ip_for_outbound_sockets=127.0.71.67
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--builtin_ntp_servers=127.0.71.84:42823,127.0.71.85:37899,127.0.71.86:36353,127.0.71.87:45495,127.0.71.88:37305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:52:55.910808 14550 heartbeater.cc:344] Connected to a master server at 127.0.71.125:44797
I20260929 01:52:55.911172 14550 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:55.912218 14550 heartbeater.cc:507] Master 127.0.71.125:44797 requested a full tablet report, sending...
I20260929 01:52:55.913761 14549 heartbeater.cc:344] Connected to a master server at 127.0.71.126:42257
I20260929 01:52:55.913975 14549 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:55.914568 14549 heartbeater.cc:507] Master 127.0.71.126:42257 requested a full tablet report, sending...
I20260929 01:52:55.914891 14548 heartbeater.cc:344] Connected to a master server at 127.0.71.124:40873
I20260929 01:52:55.915042 14548 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:55.915273 14130 ts_manager.cc:194] Registered new tserver with Master: 289825aadbca44a281341ab141d3d995 (127.0.71.66:36011)
I20260929 01:52:55.915484 14548 heartbeater.cc:507] Master 127.0.71.124:40873 requested a full tablet report, sending...
I20260929 01:52:55.915606 14046 ts_manager.cc:194] Registered new tserver with Master: 289825aadbca44a281341ab141d3d995 (127.0.71.66:36011)
I20260929 01:52:55.916546 14214 ts_manager.cc:194] Registered new tserver with Master: 289825aadbca44a281341ab141d3d995 (127.0.71.66:36011)
I20260929 01:52:55.916682 14046 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.66:51139
W20260929 01:52:56.242568 14555 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:56.242909 14555 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:56.243058 14555 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:56.251564 14555 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:56.251792 14555 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.67
I20260929 01:52:56.261937 14555 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:42823,127.0.71.85:37899,127.0.71.86:36353,127.0.71.87:45495,127.0.71.88:37305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:56.263547 14555 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:56.265419 14555 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:56.274712 14560 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:56.274909 14561 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:56.275542 14563 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:56.276273 14555 server_base.cc:1061] running on GCE node
I20260929 01:52:56.276891 14555 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:56.278046 14555 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:56.279289 14555 hybrid_clock.cc:648] HybridClock initialized: now 1790646776279151 us; error 108 us; skew 500 ppm
I20260929 01:52:56.279624 14555 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:56.282068 14555 webserver.cc:533] Webserver started at http://127.0.71.67:41423/ using document root <none> and password file <none>
I20260929 01:52:56.282804 14555 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:56.282959 14555 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:56.289466 14555 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.004s
I20260929 01:52:56.292558 14569 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:56.293958 14555 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 01:52:56.294150 14555 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/wal
uuid: "f3b2a9413c9e40d59ae47fcc88e6abf3"
format_stamp: "Formatted at 2026-09-29 01:52:40 on dist-test-slave-vz23"
I20260929 01:52:56.294848 14555 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:56.327654 14555 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:56.328509 14555 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:56.328748 14555 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:56.329648 14555 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:52:56.331462 14555 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:52:56.331588 14555 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:56.331715 14555 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:52:56.331781 14555 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:56.366943 14555 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.67:35235
I20260929 01:52:56.367007 14681 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.67:35235 every 8 connection(s)
I20260929 01:52:56.368510 14555 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
I20260929 01:52:56.372740 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 14555
I20260929 01:52:56.372972 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-2/wal/instance
I20260929 01:52:56.399590 14682 heartbeater.cc:344] Connected to a master server at 127.0.71.124:40873
I20260929 01:52:56.399766 14684 heartbeater.cc:344] Connected to a master server at 127.0.71.125:44797
I20260929 01:52:56.400019 14682 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:56.400017 14684 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:56.401212 14682 heartbeater.cc:507] Master 127.0.71.124:40873 requested a full tablet report, sending...
I20260929 01:52:56.402299 14684 heartbeater.cc:507] Master 127.0.71.125:44797 requested a full tablet report, sending...
I20260929 01:52:56.399613 14683 heartbeater.cc:344] Connected to a master server at 127.0.71.126:42257
I20260929 01:52:56.402982 14683 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:56.403441 14683 heartbeater.cc:507] Master 127.0.71.126:42257 requested a full tablet report, sending...
I20260929 01:52:56.403584 14214 ts_manager.cc:194] Registered new tserver with Master: f3b2a9413c9e40d59ae47fcc88e6abf3 (127.0.71.67:35235)
I20260929 01:52:56.403685 14130 ts_manager.cc:194] Registered new tserver with Master: f3b2a9413c9e40d59ae47fcc88e6abf3 (127.0.71.67:35235)
I20260929 01:52:56.404542 14046 ts_manager.cc:194] Registered new tserver with Master: f3b2a9413c9e40d59ae47fcc88e6abf3 (127.0.71.67:35235)
I20260929 01:52:56.405627 14046 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.67:37265
I20260929 01:52:56.406829 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:52:56.414604 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:52:56.414776 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 14015
I20260929 01:52:56.415606 14076 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:56.424161 14412 heartbeater.cc:499] Master 127.0.71.126:42257 was elected leader, sending a full tablet report...
I20260929 01:52:56.525661 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 14015
W20260929 01:52:56.541733 14117 connection.cc:573] server connection from 127.0.0.1:58886 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:52:56.542297 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:42257
--webserver_interface=127.0.71.126
--webserver_port=38495
--builtin_ntp_servers=127.0.71.84:42823,127.0.71.85:37899,127.0.71.86:36353,127.0.71.87:45495,127.0.71.88:37305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873 with env {}
W20260929 01:52:56.868528 14690 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:56.868901 14690 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:56.869010 14690 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:56.878265 14690 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:52:56.878360 14690 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:56.878465 14690 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:52:56.878556 14690 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:52:56.889210 14690 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:42823,127.0.71.85:37899,127.0.71.86:36353,127.0.71.87:45495,127.0.71.88:37305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:42257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=38495
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:56.890836 14690 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:56.892581 14690 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:56.901088 14696 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:56.901726 14695 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:56.902490 14698 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:56.902688 14690 server_base.cc:1061] running on GCE node
I20260929 01:52:56.903614 14690 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:56.905687 14690 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:56.907135 14690 hybrid_clock.cc:648] HybridClock initialized: now 1790646776906869 us; error 182 us; skew 500 ppm
I20260929 01:52:56.907593 14690 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:56.910195 14690 webserver.cc:533] Webserver started at http://127.0.71.126:38495/ using document root <none> and password file <none>
I20260929 01:52:56.910990 14690 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:56.911090 14690 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:56.916365 14690 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 01:52:56.918889 14549 heartbeater.cc:499] Master 127.0.71.126:42257 was elected leader, sending a full tablet report...
I20260929 01:52:56.920259 14704 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20260929 01:52:56.920495 14549 heartbeater.cc:646] Failed to heartbeat to 127.0.71.126:42257 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.71.126:42257: connect: Connection refused (error 111)
I20260929 01:52:56.921818 14690 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260929 01:52:56.922039 14690 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "44f4988489a442e989d3e74837e674ad"
format_stamp: "Formatted at 2026-09-29 01:52:37 on dist-test-slave-vz23"
I20260929 01:52:56.922765 14690 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:56.939270 14690 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:56.940186 14690 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:56.940547 14690 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:56.959817 14690 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:42257
I20260929 01:52:56.959825 14756 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:42257 every 8 connection(s)
I20260929 01:52:56.961580 14690 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:52:56.968999 14757 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:52:56.971508 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 14690
I20260929 01:52:56.971803 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 14274
I20260929 01:52:56.972601 14757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: Bootstrap starting.
I20260929 01:52:56.981617 14406 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:56.993013 14757 log.cc:826] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: Log is configured to *not* fsync() on all Append() calls
I20260929 01:52:57.002833 14757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:52:57.003588 14757 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: Bootstrap complete.
I20260929 01:52:57.012202 14757 raft_consensus.cc:359] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 3 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:57.013002 14757 raft_consensus.cc:740] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 3 FOLLOWER]: Becoming Follower/Learner. State: Replica: 44f4988489a442e989d3e74837e674ad, State: Initialized, Role: FOLLOWER
I20260929 01:52:57.013811 14757 consensus_queue.cc:260] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:57.014686 14763 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 3 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:57.014981 14763 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:57.015810 14757 sys_catalog.cc:565] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:52:57.027103 14774 catalog_manager.cc:1308] Loaded cluster ID: 3ed8461d403447e3800fefe76334ccef
I20260929 01:52:57.027232 14774 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: loading cluster ID for follower catalog manager: success
I20260929 01:52:57.032668 14774 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: acquiring CA information for follower catalog manager: success
I20260929 01:52:57.035961 14774 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:52:57.069198 14690 master_runner.cc:264] No leader master found from master 44f4988489a442e989d3e74837e674ad
I20260929 01:52:57.069382 14690 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:52:57.107913 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 14274
I20260929 01:52:57.127069 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:39215
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--webserver_port=34417
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:42823,127.0.71.85:37899,127.0.71.86:36353,127.0.71.87:45495,127.0.71.88:37305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:52:57.408088 14683 heartbeater.cc:499] Master 127.0.71.126:42257 was elected leader, sending a full tablet report...
I20260929 01:52:57.419644 14722 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f3b2a9413c9e40d59ae47fcc88e6abf3" instance_seqno: 1790646776358161) as {username='slave'} at 127.0.71.67:55709; Asking this server to re-register.
I20260929 01:52:57.420862 14683 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:57.422865 14722 ts_manager.cc:194] Registered new tserver with Master: f3b2a9413c9e40d59ae47fcc88e6abf3 (127.0.71.67:35235)
W20260929 01:52:57.481846 14780 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:57.482163 14780 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:57.482237 14780 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:57.490769 14780 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:57.490937 14780 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:52:57.501313 14780 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:42823,127.0.71.85:37899,127.0.71.86:36353,127.0.71.87:45495,127.0.71.88:37305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:39215
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=34417
--tserver_master_addrs=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:57.502828 14780 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:57.504587 14780 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:57.515067 14786 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:57.515038 14787 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:57.515583 14789 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:57.516466 14780 server_base.cc:1061] running on GCE node
I20260929 01:52:57.517100 14780 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:57.518215 14780 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:57.519456 14780 hybrid_clock.cc:648] HybridClock initialized: now 1790646777519299 us; error 122 us; skew 500 ppm
I20260929 01:52:57.519816 14780 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:57.521991 14780 webserver.cc:533] Webserver started at http://127.0.71.65:34417/ using document root <none> and password file <none>
I20260929 01:52:57.522745 14780 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:57.522871 14780 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:57.528771 14780 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:52:57.532358 14795 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:57.534180 14780 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:52:57.534317 14780 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "f796548a8f984835bc232e10b6ad1f76"
format_stamp: "Formatted at 2026-09-29 01:52:39 on dist-test-slave-vz23"
I20260929 01:52:57.535017 14780 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:57.550877 14780 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:57.551622 14780 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:57.551857 14780 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:57.552739 14780 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:52:57.554382 14780 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:52:57.554464 14780 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:57.554550 14780 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:52:57.554596 14780 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:57.589709 14780 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:39215
I20260929 01:52:57.589751 14907 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:39215 every 8 connection(s)
I20260929 01:52:57.591248 14780 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:52:57.601368 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 14780
I20260929 01:52:57.612646 14909 heartbeater.cc:344] Connected to a master server at 127.0.71.126:42257
I20260929 01:52:57.613130 14909 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:57.612681 14910 heartbeater.cc:344] Connected to a master server at 127.0.71.125:44797
I20260929 01:52:57.613519 14910 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:57.613955 14908 heartbeater.cc:344] Connected to a master server at 127.0.71.124:40873
I20260929 01:52:57.614115 14908 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:57.614444 14909 heartbeater.cc:507] Master 127.0.71.126:42257 requested a full tablet report, sending...
I20260929 01:52:57.614679 14908 heartbeater.cc:507] Master 127.0.71.124:40873 requested a full tablet report, sending...
I20260929 01:52:57.615411 14910 heartbeater.cc:507] Master 127.0.71.125:44797 requested a full tablet report, sending...
I20260929 01:52:57.616343 14722 ts_manager.cc:194] Registered new tserver with Master: f796548a8f984835bc232e10b6ad1f76 (127.0.71.65:39215)
I20260929 01:52:57.616830 14213 ts_manager.cc:194] Re-registered known tserver with Master: f796548a8f984835bc232e10b6ad1f76 (127.0.71.65:39215)
I20260929 01:52:57.619922 14130 ts_manager.cc:194] Re-registered known tserver with Master: f796548a8f984835bc232e10b6ad1f76 (127.0.71.65:39215)
I20260929 01:52:57.693961 14918 raft_consensus.cc:493] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44f4988489a442e989d3e74837e674ad)
I20260929 01:52:57.694337 14918 raft_consensus.cc:515] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:57.697142 14918 leader_election.cc:290] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 44f4988489a442e989d3e74837e674ad (127.0.71.126:42257), d7efd97cc2a740c0a88ee4a714d6748a (127.0.71.125:44797)
I20260929 01:52:57.698498 14140 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae67ac031b4c4b84a8b717d053cb563d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" is_pre_election: true
I20260929 01:52:57.707093 14732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae67ac031b4c4b84a8b717d053cb563d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "44f4988489a442e989d3e74837e674ad" is_pre_election: true
I20260929 01:52:57.707898 14732 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 3 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae67ac031b4c4b84a8b717d053cb563d in term 3.
I20260929 01:52:57.708626 14201 leader_election.cc:304] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44f4988489a442e989d3e74837e674ad, ae67ac031b4c4b84a8b717d053cb563d; no voters: d7efd97cc2a740c0a88ee4a714d6748a
I20260929 01:52:57.709136 14918 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 3 FOLLOWER]: Leader pre-election won for term 4
I20260929 01:52:57.709359 14918 raft_consensus.cc:493] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 3 FOLLOWER]: Starting leader election (detected failure of leader 44f4988489a442e989d3e74837e674ad)
I20260929 01:52:57.709509 14918 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 3 FOLLOWER]: Advancing to term 4
I20260929 01:52:57.712692 14918 raft_consensus.cc:515] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 4 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:57.713950 14732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae67ac031b4c4b84a8b717d053cb563d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "44f4988489a442e989d3e74837e674ad"
I20260929 01:52:57.714071 14140 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae67ac031b4c4b84a8b717d053cb563d" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "d7efd97cc2a740c0a88ee4a714d6748a"
I20260929 01:52:57.714150 14732 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 3 FOLLOWER]: Advancing to term 4
I20260929 01:52:57.714879 14918 leader_election.cc:290] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [CANDIDATE]: Term 4 election: Requested vote from peers 44f4988489a442e989d3e74837e674ad (127.0.71.126:42257), d7efd97cc2a740c0a88ee4a714d6748a (127.0.71.125:44797)
I20260929 01:52:57.719882 14732 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 4 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae67ac031b4c4b84a8b717d053cb563d in term 4.
I20260929 01:52:57.720531 14201 leader_election.cc:304] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [CANDIDATE]: Term 4 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 44f4988489a442e989d3e74837e674ad, ae67ac031b4c4b84a8b717d053cb563d; no voters: d7efd97cc2a740c0a88ee4a714d6748a
I20260929 01:52:57.720921 14918 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 4 FOLLOWER]: Leader election won for term 4
I20260929 01:52:57.721413 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 14780
I20260929 01:52:57.721433 14918 raft_consensus.cc:697] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [term 4 LEADER]: Becoming Leader. State: Replica: ae67ac031b4c4b84a8b717d053cb563d, State: Running, Role: LEADER
I20260929 01:52:57.721954 14918 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 3.6, Last appended by leader: 6, Current term: 4, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:57.725754 14920 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae67ac031b4c4b84a8b717d053cb563d. Latest consensus state: current_term: 4 leader_uuid: "ae67ac031b4c4b84a8b717d053cb563d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:57.726158 14920 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [sys.catalog]: This master's current role is: LEADER
I20260929 01:52:57.726908 14924 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:52:57.728992 14924 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:52:57.729789 14924 catalog_manager.cc:1308] Loaded cluster ID: 3ed8461d403447e3800fefe76334ccef
I20260929 01:52:57.729894 14924 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:52:57.731118 14924 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:52:57.731881 14924 catalog_manager.cc:6230] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d: Loaded TSK: 0
I20260929 01:52:57.733012 14924 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:52:57.733862 14903 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:57.822781 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 14780
I20260929 01:52:57.840071 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:39215
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--webserver_port=34417
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:42823,127.0.71.85:37899,127.0.71.86:36353,127.0.71.87:45495,127.0.71.88:37305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {}
I20260929 01:52:57.929131 14549 heartbeater.cc:344] Connected to a master server at 127.0.71.126:42257
I20260929 01:52:57.929378 14549 heartbeater.cc:499] Master 127.0.71.126:42257 was elected leader, sending a full tablet report...
I20260929 01:52:57.930371 14722 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "289825aadbca44a281341ab141d3d995" instance_seqno: 1790646775871963) as {username='slave'} at 127.0.71.66:45371; Asking this server to re-register.
I20260929 01:52:57.931070 14549 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:57.932153 14722 ts_manager.cc:194] Registered new tserver with Master: 289825aadbca44a281341ab141d3d995 (127.0.71.66:36011)
I20260929 01:52:58.027283 14928 raft_consensus.cc:493] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [term 3 FOLLOWER]: Starting pre-election (detected failure of leader 44f4988489a442e989d3e74837e674ad)
I20260929 01:52:58.027567 14928 raft_consensus.cc:515] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [term 3 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } }
I20260929 01:52:58.029601 14928 leader_election.cc:290] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [CANDIDATE]: Term 4 pre-election: Requested pre-vote from peers 44f4988489a442e989d3e74837e674ad (127.0.71.126:42257), ae67ac031b4c4b84a8b717d053cb563d (127.0.71.124:40873)
I20260929 01:52:58.037952 14732 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "44f4988489a442e989d3e74837e674ad" is_pre_election: true
I20260929 01:52:58.038375 14732 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 4 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d7efd97cc2a740c0a88ee4a714d6748a in current term 4: Already voted for candidate ae67ac031b4c4b84a8b717d053cb563d in this term.
I20260929 01:52:58.038544 14224 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" candidate_term: 4 candidate_status { last_received { term: 3 index: 6 } } ignore_live_leader: false dest_uuid: "ae67ac031b4c4b84a8b717d053cb563d" is_pre_election: true
I20260929 01:52:58.039227 14114 leader_election.cc:304] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [CANDIDATE]: Term 4 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d7efd97cc2a740c0a88ee4a714d6748a; no voters: 44f4988489a442e989d3e74837e674ad, ae67ac031b4c4b84a8b717d053cb563d
I20260929 01:52:58.039858 14928 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [term 3 FOLLOWER]: Advancing to term 4
I20260929 01:52:58.042356 14928 raft_consensus.cc:2749] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [term 4 FOLLOWER]: Leader pre-election lost for term 4. Reason: could not achieve majority
W20260929 01:52:58.182400 14926 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:58.182682 14926 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:58.182757 14926 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:52:58.182814 14926 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:58.191129 14926 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:58.191309 14926 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:52:58.201339 14926 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:42823,127.0.71.85:37899,127.0.71.86:36353,127.0.71.87:45495,127.0.71.88:37305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:39215
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=34417
--tserver_master_addrs=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:58.202848 14926 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:58.204517 14926 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:58.214111 14936 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:58.214222 14935 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:58.214704 14938 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:58.215421 14926 server_base.cc:1061] running on GCE node
I20260929 01:52:58.215967 14926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:58.217131 14926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:58.218423 14926 hybrid_clock.cc:648] HybridClock initialized: now 1790646778218223 us; error 154 us; skew 500 ppm
I20260929 01:52:58.218677 14926 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:58.220973 14926 webserver.cc:533] Webserver started at http://127.0.71.65:34417/ using document root <none> and password file <none>
I20260929 01:52:58.221617 14926 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:58.221712 14926 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:58.227180 14926 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:52:58.230176 14944 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:58.231536 14926 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:52:58.231698 14926 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "f796548a8f984835bc232e10b6ad1f76"
format_stamp: "Formatted at 2026-09-29 01:52:39 on dist-test-slave-vz23"
I20260929 01:52:58.232355 14926 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:58.249297 14926 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:58.250146 14926 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:58.250422 14926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:58.251375 14926 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:52:58.253103 14926 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:52:58.253191 14926 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:58.253346 14926 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:52:58.253434 14926 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:52:58.253808 14926 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:52:58.253945 14926 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:52:58.254020 14926 fault_injection.cc:59] injecting fault for kudu (pid 14926): FLAGS_fault_before_start (process will exit)
I20260929 01:52:58.270779 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:39215
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--webserver_port=34417
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:42823,127.0.71.85:37899,127.0.71.86:36353,127.0.71.87:45495,127.0.71.88:37305
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:52:58.310853 14732 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [term 4 FOLLOWER]: Refusing update from remote peer ae67ac031b4c4b84a8b717d053cb563d: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260929 01:52:58.311722 14953 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [LEADER]: Connected to new peer: Peer: permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 01:52:58.315603 14954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae67ac031b4c4b84a8b717d053cb563d. Latest consensus state: current_term: 4 leader_uuid: "ae67ac031b4c4b84a8b717d053cb563d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:58.315918 14954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:58.317009 14953 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "ae67ac031b4c4b84a8b717d053cb563d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:58.317329 14953 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [sys.catalog]: This master's current role is: LEADER
I20260929 01:52:58.318491 14140 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [term 4 FOLLOWER]: Refusing update from remote peer ae67ac031b4c4b84a8b717d053cb563d: Log matching property violated. Preceding OpId in replica: term: 3 index: 6. Preceding OpId from leader: term: 4 index: 7. (index mismatch)
I20260929 01:52:58.328867 14953 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 01:52:58.329814 14954 sys_catalog.cc:455] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "ae67ac031b4c4b84a8b717d053cb563d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:58.330034 14954 sys_catalog.cc:458] T 00000000000000000000000000000000 P 44f4988489a442e989d3e74837e674ad [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:58.332309 14963 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 4 leader_uuid: "ae67ac031b4c4b84a8b717d053cb563d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:58.332640 14963 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:52:58.332600 14928 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae67ac031b4c4b84a8b717d053cb563d. Latest consensus state: current_term: 4 leader_uuid: "ae67ac031b4c4b84a8b717d053cb563d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:58.332855 14928 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7efd97cc2a740c0a88ee4a714d6748a [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:52:58.619884 14952 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:52:58.620175 14952 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:52:58.620262 14952 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:52:58.628767 14952 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:52:58.628934 14952 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:52:58.638978 14952 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:42823,127.0.71.85:37899,127.0.71.86:36353,127.0.71.87:45495,127.0.71.88:37305
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:39215
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=34417
--tserver_master_addrs=127.0.71.126:42257,127.0.71.125:44797,127.0.71.124:40873
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:52:58.640661 14952 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:52:58.642400 14952 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:52:58.653023 14971 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:58.653059 14968 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:52:58.653242 14969 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:52:58.653679 14952 server_base.cc:1061] running on GCE node
I20260929 01:52:58.654280 14952 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:52:58.655470 14952 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:52:58.658802 14952 hybrid_clock.cc:648] HybridClock initialized: now 1790646778658643 us; error 118 us; skew 500 ppm
I20260929 01:52:58.659166 14952 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:52:58.661540 14952 webserver.cc:533] Webserver started at http://127.0.71.65:34417/ using document root <none> and password file <none>
I20260929 01:52:58.662328 14952 fs_manager.cc:362] Metadata directory not provided
I20260929 01:52:58.662424 14952 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:52:58.668285 14952 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:52:58.671156 14977 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:58.672700 14952 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:52:58.672861 14952 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "f796548a8f984835bc232e10b6ad1f76"
format_stamp: "Formatted at 2026-09-29 01:52:39 on dist-test-slave-vz23"
I20260929 01:52:58.673542 14952 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:52:58.719547 14952 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:52:58.720559 14952 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:52:58.720857 14952 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:52:58.721916 14952 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:52:58.723673 14952 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:52:58.723747 14952 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:58.723898 14952 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:52:58.723965 14952 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:52:58.760556 14952 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:39215
I20260929 01:52:58.760577 15089 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:39215 every 8 connection(s)
I20260929 01:52:58.762099 14952 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:52:58.762529 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 14952
I20260929 01:52:58.762873 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 14952
I20260929 01:52:58.798727 15090 heartbeater.cc:344] Connected to a master server at 127.0.71.124:40873
I20260929 01:52:58.798815 15091 heartbeater.cc:344] Connected to a master server at 127.0.71.126:42257
I20260929 01:52:58.799238 15090 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:58.799244 15091 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:58.798763 15092 heartbeater.cc:344] Connected to a master server at 127.0.71.125:44797
I20260929 01:52:58.799587 15092 heartbeater.cc:461] Registering TS with master...
I20260929 01:52:58.800683 15090 heartbeater.cc:507] Master 127.0.71.124:40873 requested a full tablet report, sending...
I20260929 01:52:58.800693 15092 heartbeater.cc:507] Master 127.0.71.125:44797 requested a full tablet report, sending...
I20260929 01:52:58.801437 14953 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 4 leader_uuid: "ae67ac031b4c4b84a8b717d053cb563d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "44f4988489a442e989d3e74837e674ad" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42257 } } peers { permanent_uuid: "d7efd97cc2a740c0a88ee4a714d6748a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44797 } } peers { permanent_uuid: "ae67ac031b4c4b84a8b717d053cb563d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40873 } } }
I20260929 01:52:58.801671 14953 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae67ac031b4c4b84a8b717d053cb563d [sys.catalog]: This master's current role is: LEADER
I20260929 01:52:58.802227 15091 heartbeater.cc:507] Master 127.0.71.126:42257 requested a full tablet report, sending...
I20260929 01:52:58.802529 14213 ts_manager.cc:194] Re-registered known tserver with Master: f796548a8f984835bc232e10b6ad1f76 (127.0.71.65:39215)
I20260929 01:52:58.803572 14722 ts_manager.cc:194] Re-registered known tserver with Master: f796548a8f984835bc232e10b6ad1f76 (127.0.71.65:39215)
I20260929 01:52:58.804098 14130 ts_manager.cc:194] Re-registered known tserver with Master: f796548a8f984835bc232e10b6ad1f76 (127.0.71.65:39215)
I20260929 01:52:58.804314 14213 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:37911
I20260929 01:52:58.805248 15085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:58.899524 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 14952
I20260929 01:52:58.919885 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 14421
I20260929 01:52:58.922510 14548 heartbeater.cc:499] Master 127.0.71.124:40873 was elected leader, sending a full tablet report...
I20260929 01:52:58.929431 14543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:59.022629 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 14421
I20260929 01:52:59.040562 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 14555
I20260929 01:52:59.054314 14677 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:59.143626 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 14555
I20260929 01:52:59.161376 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 14690
I20260929 01:52:59.167873 14752 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:59.275949 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 14690
I20260929 01:52:59.296721 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 14083
I20260929 01:52:59.298060 14160 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:59.407631 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 14083
I20260929 01:52:59.428135 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 14167
I20260929 01:52:59.429461 14244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:52:59.538166 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 14167
Shutting down hive metastore.
2026-09-29T01:52:59Z chronyd exiting
2026-09-29T01:52:59Z chronyd exiting
2026-09-29T01:52:59Z chronyd exiting
2026-09-29T01:52:59Z chronyd exiting
2026-09-29T01:52:59Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestBasicOperation/5 (39289 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/6
Loading random data
Initializing database '/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:53:00 dist-test-slave-vz23 krb5kdc[15117](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:53:00 dist-test-slave-vz23 krb5kdc[15117](info): set up 2 sockets
Sep 29 01:53:00 dist-test-slave-vz23 krb5kdc[15117](info): commencing operation
krb5kdc: starting...
W20260929 01:53:02.045291 285 mini_kdc.cc:121] Time spent starting KDC: real 2.033s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 01:53:02 dist-test-slave-vz23 krb5kdc[15117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:02.276826 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:33145
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:53:02.609354 15131 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:02.609694 15131 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:02.609859 15131 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:02.618247 15131 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:53:02.618345 15131 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:02.618402 15131 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:53:02.618508 15131 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:53:02.629223 15131 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:33145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:02.630878 15131 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:02.632251 15131 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:02.640372 15137 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:02.640456 15136 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:02.641760 15139 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:02.641870 15131 server_base.cc:1061] running on GCE node
I20260929 01:53:02.642608 15131 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:53:02.642807 15131 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:53:02.642904 15131 hybrid_clock.cc:648] HybridClock initialized: now 1790646782642881 us; error 0 us; skew 500 ppm
I20260929 01:53:02.643249 15131 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:02 dist-test-slave-vz23 krb5kdc[15117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:02.646782 15131 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:53:02.648734 15131 webserver.cc:533] Webserver started at http://127.0.71.126:45689/ using document root <none> and password file <none>
I20260929 01:53:02.649500 15131 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:02.649641 15131 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:02.649974 15131 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:53:02.652305 15131 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "4d0302fb3ffc48bca2138a0120f7c472"
format_stamp: "Formatted at 2026-09-29 01:53:02 on dist-test-slave-vz23"
I20260929 01:53:02.653138 15131 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "4d0302fb3ffc48bca2138a0120f7c472"
format_stamp: "Formatted at 2026-09-29 01:53:02 on dist-test-slave-vz23"
I20260929 01:53:02.658306 15131 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 01:53:02.661612 15145 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:02.663034 15131 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.004s
I20260929 01:53:02.663218 15131 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "4d0302fb3ffc48bca2138a0120f7c472"
format_stamp: "Formatted at 2026-09-29 01:53:02 on dist-test-slave-vz23"
I20260929 01:53:02.663451 15131 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:02.692188 15131 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:02.696400 15131 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:02.696725 15131 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:02.717983 15131 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:33145
I20260929 01:53:02.718078 15196 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:33145 every 8 connection(s)
I20260929 01:53:02.719578 15131 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:53:02.724197 15197 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:53:02.727880 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 15131
I20260929 01:53:02.728103 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:53:02.733003 15197 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:53:02 dist-test-slave-vz23 krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.125@KRBTEST.COM" created.
I20260929 01:53:02.758385 15197 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:53:02.760214 15149 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.125:34637: connect: Connection refused (error 111)
W20260929 01:53:02.762076 15197 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:34637: Network error: Client connection negotiation failed: client connection to 127.0.71.125:34637: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.125@KRBTEST.COM" created.
I20260929 01:53:02.811497 15197 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } attempt: 1
W20260929 01:53:02.813720 15197 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:34637: Network error: Client connection negotiation failed: client connection to 127.0.71.125:34637: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:02.823688 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:34637
--webserver_interface=127.0.71.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:02.882023 15197 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } attempt: 2
W20260929 01:53:02.884156 15197 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:34637: Network error: Client connection negotiation failed: client connection to 127.0.71.125:34637: connect: Connection refused (error 111)
I20260929 01:53:02.978890 15197 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } attempt: 3
W20260929 01:53:02.980952 15197 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:34637: Network error: Client connection negotiation failed: client connection to 127.0.71.125:34637: connect: Connection refused (error 111)
I20260929 01:53:03.124225 15197 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } attempt: 4
W20260929 01:53:03.126262 15197 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:34637: Network error: Client connection negotiation failed: client connection to 127.0.71.125:34637: connect: Connection refused (error 111)
W20260929 01:53:03.248772 15206 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:03.249141 15206 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:03.249291 15206 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:03.257834 15206 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:53:03.257982 15206 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:03.258088 15206 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:53:03.258168 15206 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:53:03.269104 15206 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:34637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:03.270884 15206 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:03.272658 15206 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:03.282348 15212 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:03.284215 15211 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:03.285771 15214 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:03.286644 15206 server_base.cc:1061] running on GCE node
I20260929 01:53:03.287454 15206 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:53:03.287590 15206 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:53:03.287653 15206 hybrid_clock.cc:648] HybridClock initialized: now 1790646783287637 us; error 0 us; skew 500 ppm
I20260929 01:53:03.287922 15206 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:03 dist-test-slave-vz23 krb5kdc[15117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:03.291409 15206 init.cc:377] Logged in from keytab as kudu/127.0.71.125@KRBTEST.COM (short username kudu)
I20260929 01:53:03.293548 15206 webserver.cc:533] Webserver started at http://127.0.71.125:46437/ using document root <none> and password file <none>
I20260929 01:53:03.294255 15206 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:03.294353 15206 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:03.294857 15206 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:53:03.297753 15206 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/data/instance:
uuid: "201815f40c9a4697b4bc751d4fed537d"
format_stamp: "Formatted at 2026-09-29 01:53:03 on dist-test-slave-vz23"
I20260929 01:53:03.298599 15206 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/wal/instance:
uuid: "201815f40c9a4697b4bc751d4fed537d"
format_stamp: "Formatted at 2026-09-29 01:53:03 on dist-test-slave-vz23"
I20260929 01:53:03.304021 15206 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.004s
I20260929 01:53:03.307646 15220 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:03.309242 15206 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:53:03.309438 15206 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "201815f40c9a4697b4bc751d4fed537d"
format_stamp: "Formatted at 2026-09-29 01:53:03 on dist-test-slave-vz23"
I20260929 01:53:03.309655 15206 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:03.347954 15206 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:03.352224 15206 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:03.352557 15206 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:03.373598 15206 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:34637
I20260929 01:53:03.373639 15271 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:34637 every 8 connection(s)
I20260929 01:53:03.375115 15206 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:53:03.376283 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 15206
I20260929 01:53:03.376518 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:53:03.379448 15272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:53:03.385442 15272 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.71.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.124@KRBTEST.COM" created.
Sep 29 01:53:03 dist-test-slave-vz23 krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:53:03.408555 15272 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:53:03 dist-test-slave-vz23 krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:03.425542 15197 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } attempt: 5
I20260929 01:53:03.433040 15272 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:53:03 dist-test-slave-vz23 krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
W20260929 01:53:03.434275 15224 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.124:38615: connect: Connection refused (error 111)
W20260929 01:53:03.436484 15272 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:38615: Network error: Client connection negotiation failed: client connection to 127.0.71.124:38615: connect: Connection refused (error 111)
I20260929 01:53:03.441529 15197 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:53:03.443559 15197 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:38615: Network error: Client connection negotiation failed: client connection to 127.0.71.124:38615: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.0.71.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.124@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:03.472705 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:38615
--webserver_interface=127.0.71.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:03.485880 15272 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } attempt: 1
W20260929 01:53:03.488131 15272 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:38615: Network error: Client connection negotiation failed: client connection to 127.0.71.124:38615: connect: Connection refused (error 111)
I20260929 01:53:03.494786 15197 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } attempt: 1
W20260929 01:53:03.496675 15197 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:38615: Network error: Client connection negotiation failed: client connection to 127.0.71.124:38615: connect: Connection refused (error 111)
I20260929 01:53:03.556368 15272 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } attempt: 2
W20260929 01:53:03.558679 15272 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:38615: Network error: Client connection negotiation failed: client connection to 127.0.71.124:38615: connect: Connection refused (error 111)
I20260929 01:53:03.564895 15197 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } attempt: 2
W20260929 01:53:03.566821 15197 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:38615: Network error: Client connection negotiation failed: client connection to 127.0.71.124:38615: connect: Connection refused (error 111)
I20260929 01:53:03.649964 15272 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } attempt: 3
W20260929 01:53:03.652297 15272 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:38615: Network error: Client connection negotiation failed: client connection to 127.0.71.124:38615: connect: Connection refused (error 111)
I20260929 01:53:03.673056 15197 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } attempt: 3
W20260929 01:53:03.675465 15197 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:38615: Network error: Client connection negotiation failed: client connection to 127.0.71.124:38615: connect: Connection refused (error 111)
I20260929 01:53:03.795537 15272 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } attempt: 4
W20260929 01:53:03.797822 15272 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:38615: Network error: Client connection negotiation failed: client connection to 127.0.71.124:38615: connect: Connection refused (error 111)
W20260929 01:53:03.805918 15283 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:03.806279 15283 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:03.806432 15283 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:03.814913 15283 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:53:03.815066 15283 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:03.815176 15283 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:53:03.815255 15283 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:53:03.826167 15283 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:38615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:03.827860 15283 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:03.829282 15283 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:03.837720 15288 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:03.837733 15289 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:03.838207 15291 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:03.841995 15283 server_base.cc:1061] running on GCE node
I20260929 01:53:03.842902 15283 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:53:03.843122 15283 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:53:03.843230 15283 hybrid_clock.cc:648] HybridClock initialized: now 1790646783843206 us; error 0 us; skew 500 ppm
I20260929 01:53:03.843606 15283 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:03 dist-test-slave-vz23 krb5kdc[15117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:03.847301 15283 init.cc:377] Logged in from keytab as kudu/127.0.71.124@KRBTEST.COM (short username kudu)
I20260929 01:53:03.849442 15283 webserver.cc:533] Webserver started at http://127.0.71.124:44285/ using document root <none> and password file <none>
I20260929 01:53:03.850179 15283 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:03.850317 15283 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:03.850672 15283 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:53:03.852677 15197 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } attempt: 4
I20260929 01:53:03.853286 15283 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/data/instance:
uuid: "d7f07b2f531747c4a9985e51eba98357"
format_stamp: "Formatted at 2026-09-29 01:53:03 on dist-test-slave-vz23"
I20260929 01:53:03.854187 15283 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/wal/instance:
uuid: "d7f07b2f531747c4a9985e51eba98357"
format_stamp: "Formatted at 2026-09-29 01:53:03 on dist-test-slave-vz23"
W20260929 01:53:03.854691 15197 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:38615: Network error: Client connection negotiation failed: client connection to 127.0.71.124:38615: connect: Connection refused (error 111)
I20260929 01:53:03.859506 15283 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 01:53:03.863003 15297 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:03.864348 15283 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:53:03.864626 15283 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "d7f07b2f531747c4a9985e51eba98357"
format_stamp: "Formatted at 2026-09-29 01:53:03 on dist-test-slave-vz23"
I20260929 01:53:03.864818 15283 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:03.904017 15283 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:03.908172 15283 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:03.908519 15283 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:03.929831 15283 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:38615
I20260929 01:53:03.929879 15348 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:38615 every 8 connection(s)
I20260929 01:53:03.931461 15283 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:53:03.933472 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 15283
I20260929 01:53:03.933653 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:53:03.936766 15349 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 01:53:03 dist-test-slave-vz23 krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:53:03.944983 15349 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:53:03 dist-test-slave-vz23 krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:53:03 dist-test-slave-vz23 krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:53:03.967649 15349 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:53:03 dist-test-slave-vz23 krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:53:03 dist-test-slave-vz23 krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:53:03.982992 15349 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:53:03 dist-test-slave-vz23 krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:53:04.000507 15349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357: Bootstrap starting.
I20260929 01:53:04.003945 15349 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357: Neither blocks nor log segments found. Creating new log.
I20260929 01:53:04.005007 15349 log.cc:826] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357: Log is configured to *not* fsync() on all Append() calls
I20260929 01:53:04.007912 15349 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357: No bootstrap required, opened a new log
I20260929 01:53:04.012989 15349 raft_consensus.cc:359] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } }
I20260929 01:53:04.013281 15349 raft_consensus.cc:385] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:53:04.013398 15349 raft_consensus.cc:740] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f07b2f531747c4a9985e51eba98357, State: Initialized, Role: FOLLOWER
I20260929 01:53:04.014081 15349 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } }
I20260929 01:53:04.014981 15357 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:04.015266 15357 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:04.015988 15349 sys_catalog.cc:565] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:53:04.025758 15368 catalog_manager.cc:1607] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:53:04.025892 15368 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:53:04.023072 15283 master_runner.cc:187] Error getting master registration for 127.0.71.126:33145: OK, instance_id { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" instance_seqno: 1790646782712102 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:53:04.029385 15283 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:53:04.093528 15357 raft_consensus.cc:493] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:53:04.093847 15357 raft_consensus.cc:515] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } }
I20260929 01:53:04.095498 15357 leader_election.cc:290] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 4d0302fb3ffc48bca2138a0120f7c472 (127.0.71.126:33145), 201815f40c9a4697b4bc751d4fed537d (127.0.71.125:34637)
I20260929 01:53:04.097028 15272 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } attempt: 5
Sep 29 01:53:04 dist-test-slave-vz23 krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646783, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:53:04.122732 15272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d: Bootstrap starting.
I20260929 01:53:04.126885 15272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d: Neither blocks nor log segments found. Creating new log.
I20260929 01:53:04.128206 15272 log.cc:826] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:53:04.131887 15197 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } attempt: 5
I20260929 01:53:04.137470 15272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d: No bootstrap required, opened a new log
Sep 29 01:53:04 dist-test-slave-vz23 krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646782, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:53:04.145511 15272 raft_consensus.cc:359] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } }
I20260929 01:53:04.145879 15272 raft_consensus.cc:385] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:53:04.146018 15272 raft_consensus.cc:740] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201815f40c9a4697b4bc751d4fed537d, State: Initialized, Role: FOLLOWER
I20260929 01:53:04.146827 15272 consensus_queue.cc:260] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } }
I20260929 01:53:04.148335 15371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:04.149174 15371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:04.149457 15272 sys_catalog.cc:565] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:53:04.149524 15247 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7f07b2f531747c4a9985e51eba98357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201815f40c9a4697b4bc751d4fed537d" is_pre_election: true
I20260929 01:53:04.149979 15247 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7f07b2f531747c4a9985e51eba98357 in term 0.
I20260929 01:53:04.150964 15301 leader_election.cc:304] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201815f40c9a4697b4bc751d4fed537d, d7f07b2f531747c4a9985e51eba98357; no voters:
I20260929 01:53:04.151561 15357 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:53:04.151726 15357 raft_consensus.cc:493] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:53:04.151849 15357 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:53:04.155098 15197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: Bootstrap starting.
I20260929 01:53:04.159278 15197 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: Neither blocks nor log segments found. Creating new log.
W20260929 01:53:04.160630 15303 tablet.cc:2433] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:53:04.161775 15197 log.cc:826] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: Log is configured to *not* fsync() on all Append() calls
I20260929 01:53:04.161744 15357 raft_consensus.cc:515] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } }
I20260929 01:53:04.167956 15197 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: No bootstrap required, opened a new log
I20260929 01:53:04.168162 15247 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7f07b2f531747c4a9985e51eba98357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "201815f40c9a4697b4bc751d4fed537d"
I20260929 01:53:04.168462 15247 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:53:04.175947 15197 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } }
I20260929 01:53:04.176283 15197 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:53:04.175949 15357 leader_election.cc:290] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [CANDIDATE]: Term 1 election: Requested vote from peers 4d0302fb3ffc48bca2138a0120f7c472 (127.0.71.126:33145), 201815f40c9a4697b4bc751d4fed537d (127.0.71.125:34637)
I20260929 01:53:04.176434 15197 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0302fb3ffc48bca2138a0120f7c472, State: Initialized, Role: FOLLOWER
I20260929 01:53:04.177645 15197 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } }
I20260929 01:53:04.177737 15247 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7f07b2f531747c4a9985e51eba98357 in term 1.
I20260929 01:53:04.178918 15301 leader_election.cc:304] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 201815f40c9a4697b4bc751d4fed537d, d7f07b2f531747c4a9985e51eba98357; no voters:
I20260929 01:53:04.179623 15357 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:53:04.188815 15357 raft_consensus.cc:697] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [term 1 LEADER]: Becoming Leader. State: Replica: d7f07b2f531747c4a9985e51eba98357, State: Running, Role: LEADER
I20260929 01:53:04.191082 15357 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } }
I20260929 01:53:04.202221 15197 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:53:04.202291 15171 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7f07b2f531747c4a9985e51eba98357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0302fb3ffc48bca2138a0120f7c472"
I20260929 01:53:04.202852 15171 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:53:04.207731 15171 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7f07b2f531747c4a9985e51eba98357 in term 1.
I20260929 01:53:04.209070 15172 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7f07b2f531747c4a9985e51eba98357" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "4d0302fb3ffc48bca2138a0120f7c472" is_pre_election: true
I20260929 01:53:04.209522 15172 raft_consensus.cc:2376] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d7f07b2f531747c4a9985e51eba98357 in term 1. Re-sending same reply.
I20260929 01:53:04.201365 15376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:04.210848 15376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:04.217887 15378 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7f07b2f531747c4a9985e51eba98357. Latest consensus state: current_term: 1 leader_uuid: "d7f07b2f531747c4a9985e51eba98357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:04.218341 15378 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [sys.catalog]: This master's current role is: LEADER
I20260929 01:53:04.224061 15384 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:53:04.227283 15384 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:53:04.244807 15247 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [term 1 FOLLOWER]: Refusing update from remote peer d7f07b2f531747c4a9985e51eba98357: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:53:04.246126 15378 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:53:04.254400 15171 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [term 1 FOLLOWER]: Refusing update from remote peer d7f07b2f531747c4a9985e51eba98357: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:53:04.255630 15378 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:53:04.281152 15371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7f07b2f531747c4a9985e51eba98357. Latest consensus state: current_term: 1 leader_uuid: "d7f07b2f531747c4a9985e51eba98357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:04.281447 15371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:04.283794 15402 catalog_manager.cc:818] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 01:53:04.285459 15376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7f07b2f531747c4a9985e51eba98357. Latest consensus state: current_term: 1 leader_uuid: "d7f07b2f531747c4a9985e51eba98357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:04.285763 15376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:04.288029 15385 mvcc.cc:204] Tried to move back new op lower bound from 7334489228251422720 to 7334489228069691392. Current Snapshot: MvccSnapshot[applied={T|T < 7334489228251422720}]
I20260929 01:53:04.288200 15206 master_runner.cc:187] Error getting master registration for 127.0.71.126:33145: OK, instance_id { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" instance_seqno: 1790646782712102 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:53:04.300137 15206 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:53:04.301342 15403 catalog_manager.cc:818] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 01:53:04.304734 15376 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d7f07b2f531747c4a9985e51eba98357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:04.305081 15376 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:04.308984 15378 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d7f07b2f531747c4a9985e51eba98357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:04.309362 15378 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [sys.catalog]: This master's current role is: LEADER
I20260929 01:53:04.309659 15378 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d7f07b2f531747c4a9985e51eba98357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:04.309890 15378 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [sys.catalog]: This master's current role is: LEADER
I20260929 01:53:04.311030 15371 sys_catalog.cc:455] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "d7f07b2f531747c4a9985e51eba98357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:04.311331 15371 sys_catalog.cc:458] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:04.312033 15384 catalog_manager.cc:1396] Generated new cluster ID: 71984c3cb0d0405c972b708a17c5de2e
I20260929 01:53:04.312157 15384 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:53:04.340545 15384 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 01:53:04.341928 15384 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:53:04.353259 15384 catalog_manager.cc:6219] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357: Generated new TSK 0
I20260929 01:53:04.354355 15384 catalog_manager.cc:1563] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:04.460774 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:53:04.801079 15413 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:04.801401 15413 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:04.801563 15413 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:04.810123 15413 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:04.810434 15413 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:53:04.821197 15413 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:04.822726 15413 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:04.823999 15413 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:04.833931 15419 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:04.834054 15418 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:04.834205 15421 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:04.834838 15413 server_base.cc:1061] running on GCE node
I20260929 01:53:04.835438 15413 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:53:04.835573 15413 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:53:04.835638 15413 hybrid_clock.cc:648] HybridClock initialized: now 1790646784835617 us; error 0 us; skew 500 ppm
I20260929 01:53:04.835878 15413 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:04 dist-test-slave-vz23 krb5kdc[15117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:04.846519 15413 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:53:04.848357 15413 webserver.cc:533] Webserver started at http://127.0.71.65:45339/ using document root <none> and password file <none>
I20260929 01:53:04.849170 15413 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:04.849316 15413 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:04.849710 15413 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:53:04.852237 15413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "25071766fd4a4a0e8c96d4a1df800197"
format_stamp: "Formatted at 2026-09-29 01:53:04 on dist-test-slave-vz23"
I20260929 01:53:04.853569 15413 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "25071766fd4a4a0e8c96d4a1df800197"
format_stamp: "Formatted at 2026-09-29 01:53:04 on dist-test-slave-vz23"
I20260929 01:53:04.860229 15413 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260929 01:53:04.863885 15427 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:04.865587 15413 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:53:04.865768 15413 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "25071766fd4a4a0e8c96d4a1df800197"
format_stamp: "Formatted at 2026-09-29 01:53:04 on dist-test-slave-vz23"
I20260929 01:53:04.866003 15413 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:04.887209 15413 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:04.891443 15413 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:04.891736 15413 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:04.892778 15413 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:04.894423 15413 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:04.894527 15413 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:04.894672 15413 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:04.894748 15413 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:04.940258 15413 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:44311
I20260929 01:53:04.940292 15539 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:44311 every 8 connection(s)
I20260929 01:53:04.941728 15413 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:53:04.942480 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 15413
I20260929 01:53:04.942759 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
WARNING: no policy specified for kudu/127.0.71.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.66@KRBTEST.COM" created.
Sep 29 01:53:04 dist-test-slave-vz23 krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:53:04 dist-test-slave-vz23 krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:53:04 dist-test-slave-vz23 krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646784, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:53:04.993901 15541 heartbeater.cc:344] Connected to a master server at 127.0.71.126:33145
I20260929 01:53:04.994341 15541 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:04.995561 15541 heartbeater.cc:507] Master 127.0.71.126:33145 requested a full tablet report, sending...
I20260929 01:53:04.998567 15161 ts_manager.cc:194] Registered new tserver with Master: 25071766fd4a4a0e8c96d4a1df800197 (127.0.71.65:44311)
I20260929 01:53:05.001673 15544 heartbeater.cc:344] Connected to a master server at 127.0.71.125:34637
I20260929 01:53:05.001869 15544 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:05.002502 15544 heartbeater.cc:507] Master 127.0.71.125:34637 requested a full tablet report, sending...
I20260929 01:53:05.004849 15236 ts_manager.cc:194] Registered new tserver with Master: 25071766fd4a4a0e8c96d4a1df800197 (127.0.71.65:44311)
I20260929 01:53:05.012580 15540 heartbeater.cc:344] Connected to a master server at 127.0.71.124:38615
I20260929 01:53:05.012785 15540 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:05.013482 15540 heartbeater.cc:507] Master 127.0.71.124:38615 requested a full tablet report, sending...
I20260929 01:53:05.015496 15314 ts_manager.cc:194] Registered new tserver with Master: 25071766fd4a4a0e8c96d4a1df800197 (127.0.71.65:44311)
I20260929 01:53:05.017349 15314 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:32815
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:05.065969 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.66:0
--local_ip_for_outbound_sockets=127.0.71.66
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:05.288995 15402 catalog_manager.cc:1308] Loaded cluster ID: 71984c3cb0d0405c972b708a17c5de2e
I20260929 01:53:05.289171 15402 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d: loading cluster ID for follower catalog manager: success
I20260929 01:53:05.293151 15402 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d: acquiring CA information for follower catalog manager: success
I20260929 01:53:05.295687 15402 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:53:05.308969 15403 catalog_manager.cc:1308] Loaded cluster ID: 71984c3cb0d0405c972b708a17c5de2e
I20260929 01:53:05.309190 15403 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: loading cluster ID for follower catalog manager: success
I20260929 01:53:05.313153 15403 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: acquiring CA information for follower catalog manager: success
I20260929 01:53:05.315936 15403 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 01:53:05.459093 15554 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:05.459553 15554 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:05.459707 15554 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:05.473742 15554 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:05.474031 15554 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.66
I20260929 01:53:05.491820 15554 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.71.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:05.494166 15554 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:05.495880 15554 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:05.507203 15563 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:05.507969 15564 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:05.519253 15566 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:05.519797 15554 server_base.cc:1061] running on GCE node
I20260929 01:53:05.520596 15554 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:53:05.520817 15554 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:53:05.520939 15554 hybrid_clock.cc:648] HybridClock initialized: now 1790646785520909 us; error 0 us; skew 500 ppm
I20260929 01:53:05.521287 15554 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:05 dist-test-slave-vz23 krb5kdc[15117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:05.525172 15554 init.cc:377] Logged in from keytab as kudu/127.0.71.66@KRBTEST.COM (short username kudu)
I20260929 01:53:05.527356 15554 webserver.cc:533] Webserver started at http://127.0.71.66:39951/ using document root <none> and password file <none>
I20260929 01:53:05.528164 15554 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:05.528313 15554 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:05.528738 15554 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:53:05.531375 15554 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/data/instance:
uuid: "d3ab5a94294f4deab6f57b2fa18d9728"
format_stamp: "Formatted at 2026-09-29 01:53:05 on dist-test-slave-vz23"
I20260929 01:53:05.532294 15554 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/wal/instance:
uuid: "d3ab5a94294f4deab6f57b2fa18d9728"
format_stamp: "Formatted at 2026-09-29 01:53:05 on dist-test-slave-vz23"
I20260929 01:53:05.538515 15554 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 01:53:05.542363 15572 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:05.544119 15554 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:53:05.544302 15554 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/wal
uuid: "d3ab5a94294f4deab6f57b2fa18d9728"
format_stamp: "Formatted at 2026-09-29 01:53:05 on dist-test-slave-vz23"
I20260929 01:53:05.544536 15554 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:05.589107 15554 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:05.593623 15554 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:05.593958 15554 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:05.595171 15554 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:05.597072 15554 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:05.597183 15554 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:05.597325 15554 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:05.597411 15554 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:05.646415 15554 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.66:45259
I20260929 01:53:05.646508 15684 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.66:45259 every 8 connection(s)
I20260929 01:53:05.648399 15554 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
I20260929 01:53:05.656002 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 15554
I20260929 01:53:05.656275 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/wal/instance
WARNING: no policy specified for kudu/127.0.71.67@KRBTEST.COM; defaulting to no policy
Sep 29 01:53:05 dist-test-slave-vz23 krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.67@KRBTEST.COM" created.
Sep 29 01:53:05 dist-test-slave-vz23 krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:53:05.771528 15686 heartbeater.cc:344] Connected to a master server at 127.0.71.126:33145
I20260929 01:53:05.771962 15686 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:05.773111 15686 heartbeater.cc:507] Master 127.0.71.126:33145 requested a full tablet report, sending...
I20260929 01:53:05.775177 15161 ts_manager.cc:194] Registered new tserver with Master: d3ab5a94294f4deab6f57b2fa18d9728 (127.0.71.66:45259)
I20260929 01:53:05.782598 15685 heartbeater.cc:344] Connected to a master server at 127.0.71.124:38615
I20260929 01:53:05.782720 15685 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:05.783253 15685 heartbeater.cc:507] Master 127.0.71.124:38615 requested a full tablet report, sending...
I20260929 01:53:05.784497 15314 ts_manager.cc:194] Registered new tserver with Master: d3ab5a94294f4deab6f57b2fa18d9728 (127.0.71.66:45259)
I20260929 01:53:05.785517 15314 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.66@KRBTEST.COM'} at 127.0.71.66:33213
Sep 29 01:53:05 dist-test-slave-vz23 krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646785, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:53:05.797079 15690 heartbeater.cc:344] Connected to a master server at 127.0.71.125:34637
I20260929 01:53:05.797241 15690 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:05.797602 15690 heartbeater.cc:507] Master 127.0.71.125:34637 requested a full tablet report, sending...
I20260929 01:53:05.798792 15236 ts_manager.cc:194] Registered new tserver with Master: d3ab5a94294f4deab6f57b2fa18d9728 (127.0.71.66:45259)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
W20260929 01:53:05.803378 285 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.0.71.67: real 0.140s user 0.002s sys 0.017s
WARNING: no policy specified for HTTP/127.0.71.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:05.860970 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.67:0
--local_ip_for_outbound_sockets=127.0.71.67
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:06.020345 15540 heartbeater.cc:499] Master 127.0.71.124:38615 was elected leader, sending a full tablet report...
W20260929 01:53:06.198777 15699 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:06.199137 15699 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:06.199306 15699 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:06.207556 15699 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:06.207810 15699 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.67
I20260929 01:53:06.218698 15699 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.71.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:06.220403 15699 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:06.221767 15699 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:06.231560 15704 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:06.231632 15705 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:06.232157 15707 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:06.233210 15699 server_base.cc:1061] running on GCE node
I20260929 01:53:06.233872 15699 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:53:06.234077 15699 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:53:06.234179 15699 hybrid_clock.cc:648] HybridClock initialized: now 1790646786234146 us; error 0 us; skew 500 ppm
I20260929 01:53:06.234436 15699 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:06 dist-test-slave-vz23 krb5kdc[15117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:06.238008 15699 init.cc:377] Logged in from keytab as kudu/127.0.71.67@KRBTEST.COM (short username kudu)
I20260929 01:53:06.239661 15699 webserver.cc:533] Webserver started at http://127.0.71.67:35189/ using document root <none> and password file <none>
I20260929 01:53:06.240412 15699 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:06.240571 15699 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:06.240867 15699 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:53:06.243227 15699 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/data/instance:
uuid: "4a4de4f4a1c4419998114ad18ced1014"
format_stamp: "Formatted at 2026-09-29 01:53:06 on dist-test-slave-vz23"
I20260929 01:53:06.244016 15699 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/wal/instance:
uuid: "4a4de4f4a1c4419998114ad18ced1014"
format_stamp: "Formatted at 2026-09-29 01:53:06 on dist-test-slave-vz23"
I20260929 01:53:06.249773 15699 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 01:53:06.253295 15713 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:06.254606 15699 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:53:06.254740 15699 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/wal
uuid: "4a4de4f4a1c4419998114ad18ced1014"
format_stamp: "Formatted at 2026-09-29 01:53:06 on dist-test-slave-vz23"
I20260929 01:53:06.254902 15699 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:06.278450 15699 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:06.283290 15699 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:06.283627 15699 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:06.284763 15699 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:06.286268 15699 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:06.286353 15699 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:06.286453 15699 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:06.286495 15699 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:06.322005 15699 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.67:46651
I20260929 01:53:06.322086 15825 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.67:46651 every 8 connection(s)
I20260929 01:53:06.323441 15699 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
I20260929 01:53:06.324213 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 15699
I20260929 01:53:06.324488 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/wal/instance
Sep 29 01:53:06 dist-test-slave-vz23 krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:53:06 dist-test-slave-vz23 krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:53:06 dist-test-slave-vz23 krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:53:06.358673 15826 heartbeater.cc:344] Connected to a master server at 127.0.71.124:38615
I20260929 01:53:06.358834 15828 heartbeater.cc:344] Connected to a master server at 127.0.71.125:34637
I20260929 01:53:06.359119 15828 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:06.359122 15826 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:06.359196 15827 heartbeater.cc:344] Connected to a master server at 127.0.71.126:33145
I20260929 01:53:06.359373 15827 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:06.360313 15828 heartbeater.cc:507] Master 127.0.71.125:34637 requested a full tablet report, sending...
I20260929 01:53:06.360353 15826 heartbeater.cc:507] Master 127.0.71.124:38615 requested a full tablet report, sending...
I20260929 01:53:06.360378 15827 heartbeater.cc:507] Master 127.0.71.126:33145 requested a full tablet report, sending...
I20260929 01:53:06.362078 15314 ts_manager.cc:194] Registered new tserver with Master: 4a4de4f4a1c4419998114ad18ced1014 (127.0.71.67:46651)
I20260929 01:53:06.362164 15161 ts_manager.cc:194] Registered new tserver with Master: 4a4de4f4a1c4419998114ad18ced1014 (127.0.71.67:46651)
I20260929 01:53:06.362807 15236 ts_manager.cc:194] Registered new tserver with Master: 4a4de4f4a1c4419998114ad18ced1014 (127.0.71.67:46651)
I20260929 01:53:06.363305 15314 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.67@KRBTEST.COM'} at 127.0.71.67:35265
I20260929 01:53:06.370781 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:53:06.378170 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:53:06.378376 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 15131
I20260929 01:53:06.379302 15192 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:53:06.487982 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 15131
I20260929 01:53:06.509442 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:33145
--webserver_interface=127.0.71.126
--webserver_port=45689
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:53:06.778360 15686 heartbeater.cc:646] Failed to heartbeat to 127.0.71.126:33145 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.71.126:33145: connect: Connection refused (error 111)
I20260929 01:53:06.788252 15685 heartbeater.cc:499] Master 127.0.71.124:38615 was elected leader, sending a full tablet report...
W20260929 01:53:06.845830 15298 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.126:33145: connect: Connection refused (error 111)
W20260929 01:53:06.847374 15298 consensus_peers.cc:597] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 -> Peer 4d0302fb3ffc48bca2138a0120f7c472 (127.0.71.126:33145): Couldn't send request to peer 4d0302fb3ffc48bca2138a0120f7c472. Status: Network error: Client connection negotiation failed: client connection to 127.0.71.126:33145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260929 01:53:06.889545 15837 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:06.889901 15837 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:06.890096 15837 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:06.899727 15837 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:53:06.899849 15837 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:06.899945 15837 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:53:06.900038 15837 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:53:06.911593 15837 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:33145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=45689
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:06.913368 15837 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:06.914754 15837 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:06.923111 15845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:06.923203 15846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:06.924805 15848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:06.925415 15837 server_base.cc:1061] running on GCE node
I20260929 01:53:06.926100 15837 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:53:06.926285 15837 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:53:06.926366 15837 hybrid_clock.cc:648] HybridClock initialized: now 1790646786926344 us; error 0 us; skew 500 ppm
I20260929 01:53:06.926649 15837 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:06 dist-test-slave-vz23 krb5kdc[15117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:06.930506 15837 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:53:06.932649 15837 webserver.cc:533] Webserver started at http://127.0.71.126:45689/ using document root <none> and password file <none>
I20260929 01:53:06.933395 15837 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:06.933552 15837 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:06.939304 15837 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:53:06.942725 15854 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:06.944104 15837 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:53:06.944293 15837 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "4d0302fb3ffc48bca2138a0120f7c472"
format_stamp: "Formatted at 2026-09-29 01:53:02 on dist-test-slave-vz23"
I20260929 01:53:06.945075 15837 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:06.962270 15837 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:06.966507 15837 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:06.966785 15837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:06.988895 15837 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:33145
I20260929 01:53:06.988948 15905 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:33145 every 8 connection(s)
I20260929 01:53:06.990589 15837 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:53:06.998018 15906 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:53:07.000990 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 15837
I20260929 01:53:07.001391 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 15413
I20260929 01:53:07.001952 15906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: Bootstrap starting.
Sep 29 01:53:07 dist-test-slave-vz23 krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646782, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 01:53:07.030807 15906 log.cc:826] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: Log is configured to *not* fsync() on all Append() calls
I20260929 01:53:07.033424 15535 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:53:07.043768 15906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:53:07.044662 15906 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: Bootstrap complete.
I20260929 01:53:07.055884 15906 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } }
I20260929 01:53:07.057073 15906 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0302fb3ffc48bca2138a0120f7c472, State: Initialized, Role: FOLLOWER
I20260929 01:53:07.057991 15906 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } }
I20260929 01:53:07.058970 15914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:07.059258 15914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:07.060670 15906 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:53:07.081952 15925 catalog_manager.cc:1308] Loaded cluster ID: 71984c3cb0d0405c972b708a17c5de2e
I20260929 01:53:07.082798 15925 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: loading cluster ID for follower catalog manager: success
I20260929 01:53:07.087010 15925 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: acquiring CA information for follower catalog manager: success
I20260929 01:53:07.090461 15925 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Sep 29 01:53:07 dist-test-slave-vz23 krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646786, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:53:07.181674 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 15413
I20260929 01:53:07.199223 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:44311
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--webserver_port=45339
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:53:07.200062 15911 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:07.009695 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:34179 (local address 127.0.71.126:33145)
0929 01:53:07.020408 (+ 10713us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:07.020426 (+ 18us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:07.020893 (+ 467us) server_negotiation.cc:408] Connection header received
0929 01:53:07.023014 (+ 2121us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:53:07.023051 (+ 37us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:53:07.023724 (+ 673us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:53:07.024306 (+ 582us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:53:07.026163 (+ 1857us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:53:07.027785 (+ 1622us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:53:07.029163 (+ 1378us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:53:07.029760 (+ 597us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0929 01:53:07.031441 (+ 1681us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0929 01:53:07.031535 (+ 94us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0929 01:53:07.031559 (+ 24us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0929 01:53:07.031611 (+ 52us) server_negotiation.cc:923] calling sasl_server_start()
0929 01:53:07.036693 (+ 5082us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0929 01:53:07.181146 (+144453us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0929 01:53:07.181156 (+ 10us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0929 01:53:07.181160 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0929 01:53:07.181293 (+ 133us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0929 01:53:07.181719 (+ 426us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0929 01:53:07.181725 (+ 6us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0929 01:53:07.181729 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0929 01:53:07.182094 (+ 365us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0929 01:53:07.182239 (+ 145us) server_negotiation.cc:1036] Waiting for connection context
0929 01:53:07.198525 (+ 16286us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:34179: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":10455,"thread_start_us":10223,"threads_started":1}
I20260929 01:53:07.365849 15826 heartbeater.cc:499] Master 127.0.71.124:38615 was elected leader, sending a full tablet report...
I20260929 01:53:07.376123 15871 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "4a4de4f4a1c4419998114ad18ced1014" instance_seqno: 1790646786313760) as {username='kudu', principal='kudu/127.0.71.67@KRBTEST.COM'} at 127.0.71.67:50949; Asking this server to re-register.
I20260929 01:53:07.377127 15827 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:07.378837 15871 ts_manager.cc:194] Registered new tserver with Master: 4a4de4f4a1c4419998114ad18ced1014 (127.0.71.67:46651)
I20260929 01:53:07.380648 15557 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d7f07b2f531747c4a9985e51eba98357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:07.380956 15557 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [sys.catalog]: This master's current role is: LEADER
I20260929 01:53:07.424371 15914 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7f07b2f531747c4a9985e51eba98357. Latest consensus state: current_term: 1 leader_uuid: "d7f07b2f531747c4a9985e51eba98357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:07.424711 15914 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:53:07.598277 15931 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:07.598600 15931 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:07.598798 15931 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:07.607949 15931 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:07.608210 15931 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:53:07.619325 15931 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:44311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=45339
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:07.621074 15931 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:07.622396 15931 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:07.632184 15942 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:07.632169 15941 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:07.633317 15944 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:07.633538 15931 server_base.cc:1061] running on GCE node
I20260929 01:53:07.634126 15931 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:53:07.634302 15931 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:53:07.634387 15931 hybrid_clock.cc:648] HybridClock initialized: now 1790646787634370 us; error 0 us; skew 500 ppm
I20260929 01:53:07.634708 15931 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:07 dist-test-slave-vz23 krb5kdc[15117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:07.638247 15931 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:53:07.639855 15931 webserver.cc:533] Webserver started at http://127.0.71.65:45339/ using document root <none> and password file <none>
I20260929 01:53:07.640600 15931 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:07.640708 15931 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:07.646224 15931 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:53:07.649060 15950 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:07.650609 15931 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:53:07.650770 15931 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "25071766fd4a4a0e8c96d4a1df800197"
format_stamp: "Formatted at 2026-09-29 01:53:04 on dist-test-slave-vz23"
I20260929 01:53:07.651494 15931 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:07.670969 15931 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:07.674767 15931 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:07.675045 15931 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:07.676003 15931 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:07.677560 15931 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:07.677673 15931 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:07.677805 15931 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:07.677888 15931 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:07.713753 15931 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:44311
I20260929 01:53:07.713864 16062 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:44311 every 8 connection(s)
I20260929 01:53:07.715353 15931 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:53:07.716111 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 15931
Sep 29 01:53:07 dist-test-slave-vz23 krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:53:07 dist-test-slave-vz23 krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:53:07 dist-test-slave-vz23 krb5kdc[15117](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646787, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:53:07.753587 16064 heartbeater.cc:344] Connected to a master server at 127.0.71.126:33145
I20260929 01:53:07.753933 16063 heartbeater.cc:344] Connected to a master server at 127.0.71.124:38615
I20260929 01:53:07.754052 16064 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:07.754145 16063 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:07.755340 16063 heartbeater.cc:507] Master 127.0.71.124:38615 requested a full tablet report, sending...
I20260929 01:53:07.755364 16064 heartbeater.cc:507] Master 127.0.71.126:33145 requested a full tablet report, sending...
I20260929 01:53:07.755441 16065 heartbeater.cc:344] Connected to a master server at 127.0.71.125:34637
I20260929 01:53:07.755594 16065 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:07.756196 16065 heartbeater.cc:507] Master 127.0.71.125:34637 requested a full tablet report, sending...
I20260929 01:53:07.757465 15871 ts_manager.cc:194] Registered new tserver with Master: 25071766fd4a4a0e8c96d4a1df800197 (127.0.71.65:44311)
I20260929 01:53:07.757670 15236 ts_manager.cc:194] Re-registered known tserver with Master: 25071766fd4a4a0e8c96d4a1df800197 (127.0.71.65:44311)
I20260929 01:53:07.758252 15314 ts_manager.cc:194] Re-registered known tserver with Master: 25071766fd4a4a0e8c96d4a1df800197 (127.0.71.65:44311)
I20260929 01:53:07.759155 15314 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:40553
I20260929 01:53:07.765916 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 15931
W20260929 01:53:07.766398 16075 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:07.762841 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:44928 (local address 127.0.71.65:44311)
0929 01:53:07.763527 (+ 686us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:07.763533 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:07.763556 (+ 23us) server_negotiation.cc:408] Connection header received
0929 01:53:07.763600 (+ 44us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:53:07.763606 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:53:07.763775 (+ 169us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:53:07.764047 (+ 272us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:53:07.765345 (+ 1298us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44928: BlockingRecv error: recv got EOF from 127.0.0.1:44928 (error 108)
Metrics: {"server-negotiator.queue_time_us":537,"thread_start_us":349,"threads_started":1}
W20260929 01:53:07.767910 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 15931: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:44311: server requires authentication, but client does not have Kerberos credentials available
W20260929 01:53:07.768066 16075 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:07.766586 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:44934 (local address 127.0.71.65:44311)
0929 01:53:07.766916 (+ 330us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:07.766922 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:07.766944 (+ 22us) server_negotiation.cc:408] Connection header received
0929 01:53:07.766996 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:53:07.766999 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:53:07.767046 (+ 47us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:53:07.767119 (+ 73us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:53:07.767841 (+ 722us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44934: BlockingRecv error: recv got EOF from 127.0.0.1:44934 (error 108)
Metrics: {"server-negotiator.queue_time_us":204,"thread_start_us":292,"threads_started":1}
I20260929 01:53:07.768095 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 15931
I20260929 01:53:07.785585 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:44311
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--webserver_port=45339
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:07.786523 15686 heartbeater.cc:344] Connected to a master server at 127.0.71.126:33145
I20260929 01:53:07.787602 15871 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d3ab5a94294f4deab6f57b2fa18d9728" instance_seqno: 1790646785635915) as {username='kudu', principal='kudu/127.0.71.66@KRBTEST.COM'} at 127.0.71.66:43347; Asking this server to re-register.
I20260929 01:53:07.788152 15686 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:07.789281 15871 ts_manager.cc:194] Registered new tserver with Master: d3ab5a94294f4deab6f57b2fa18d9728 (127.0.71.66:45259)
I20260929 01:53:07.952878 15934 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "d7f07b2f531747c4a9985e51eba98357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:07.953178 15934 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [sys.catalog]: This master's current role is: LEADER
W20260929 01:53:08.128130 16078 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:08.128428 16078 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:08.128599 16078 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:53:08.128662 16078 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:08.137364 16078 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:08.137575 16078 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:53:08.148068 16078 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:44311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=45339
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:08.149838 16078 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:08.151208 16078 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:08.160844 16086 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:08.161144 16087 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:08.161314 16089 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:08.162405 16078 server_base.cc:1061] running on GCE node
I20260929 01:53:08.162949 16078 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:53:08.163136 16078 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:53:08.163249 16078 hybrid_clock.cc:648] HybridClock initialized: now 1790646788163211 us; error 0 us; skew 500 ppm
I20260929 01:53:08.163553 16078 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:08 dist-test-slave-vz23 krb5kdc[15117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:08.166867 16078 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:53:08.168634 16078 webserver.cc:533] Webserver started at http://127.0.71.65:45339/ using document root <none> and password file <none>
I20260929 01:53:08.169431 16078 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:08.169548 16078 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:08.174973 16078 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:53:08.177811 16095 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:08.179173 16078 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:53:08.179355 16078 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "25071766fd4a4a0e8c96d4a1df800197"
format_stamp: "Formatted at 2026-09-29 01:53:04 on dist-test-slave-vz23"
I20260929 01:53:08.180080 16078 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:08.197846 16078 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:08.201572 16078 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:08.201877 16078 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:08.202831 16078 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:08.204409 16078 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:08.204547 16078 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 01:53:08.204699 16078 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:08.204772 16078 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:53:08.205168 16078 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:53:08.205253 16078 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:53:08.205331 16078 fault_injection.cc:59] injecting fault for kudu (pid 16078): FLAGS_fault_before_start (process will exit)
I20260929 01:53:08.228488 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:44311
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--webserver_port=45339
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:53:08.563086 16103 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:08.563441 16103 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:08.563596 16103 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:08.572148 16103 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:08.572394 16103 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:53:08.583133 16103 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:44311
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=45339
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:08.584836 16103 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:08.586169 16103 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:08.595966 16108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:08.595995 16109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:08.596662 16111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:08.597131 16103 server_base.cc:1061] running on GCE node
I20260929 01:53:08.597693 16103 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:53:08.597880 16103 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:53:08.597985 16103 hybrid_clock.cc:648] HybridClock initialized: now 1790646788597962 us; error 0 us; skew 500 ppm
I20260929 01:53:08.598336 16103 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:08 dist-test-slave-vz23 krb5kdc[15117](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646788, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:08.601655 16103 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:53:08.603284 16103 webserver.cc:533] Webserver started at http://127.0.71.65:45339/ using document root <none> and password file <none>
I20260929 01:53:08.604013 16103 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:08.604146 16103 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:08.609432 16103 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:53:08.612107 16117 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:08.613561 16103 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:53:08.613727 16103 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "25071766fd4a4a0e8c96d4a1df800197"
format_stamp: "Formatted at 2026-09-29 01:53:04 on dist-test-slave-vz23"
I20260929 01:53:08.614445 16103 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:08.640836 16103 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:08.644589 16103 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:08.644860 16103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:08.646047 16103 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:08.647590 16103 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:08.647709 16103 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:08.647840 16103 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:08.647935 16103 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:08.682821 16103 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:44311
I20260929 01:53:08.682905 16229 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:44311 every 8 connection(s)
I20260929 01:53:08.684353 16103 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:53:08.690110 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 16103
I20260929 01:53:08.690533 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 16103
W20260929 01:53:08.708885 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 16103: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:44311: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:53:08.709088 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 16103
I20260929 01:53:08.728585 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 15554
W20260929 01:53:08.728695 16235 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:08.687757 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:59791 (local address 127.0.71.126:33145)
0929 01:53:08.703422 (+ 15665us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:08.703430 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:08.704202 (+ 772us) server_negotiation.cc:408] Connection header received
0929 01:53:08.706892 (+ 2690us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:53:08.706902 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:53:08.707031 (+ 129us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:53:08.707163 (+ 132us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:53:08.728298 (+ 21135us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:59791: BlockingRecv error: recv got EOF from 127.0.71.65:59791 (error 108)
Metrics: {"server-negotiator.queue_time_us":15562,"thread_start_us":15447,"threads_started":1}
W20260929 01:53:08.729349 16241 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:08.705667 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:37243 (local address 127.0.71.124:38615)
0929 01:53:08.712912 (+ 7245us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:08.712917 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:08.712936 (+ 19us) server_negotiation.cc:408] Connection header received
0929 01:53:08.728363 (+ 15427us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:37243: BlockingRecv error: recv got EOF from 127.0.71.65:37243 (error 108)
Metrics: {"server-negotiator.queue_time_us":7146,"thread_start_us":7041,"threads_started":1}
W20260929 01:53:08.729346 16234 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:08.687613 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:47611 (local address 127.0.71.125:34637)
0929 01:53:08.703045 (+ 15432us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:08.703055 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:08.704725 (+ 1670us) server_negotiation.cc:408] Connection header received
0929 01:53:08.706869 (+ 2144us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:53:08.706877 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:53:08.706967 (+ 90us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:53:08.707087 (+ 120us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:53:08.728420 (+ 21333us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:47611: BlockingRecv error: recv got EOF from 127.0.71.65:47611 (error 108)
Metrics: {"server-negotiator.queue_time_us":15256,"thread_start_us":15707,"threads_started":1}
W20260929 01:53:08.731408 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 15554: Not authorized: Client connection negotiation failed: client connection to 127.0.71.66:45259: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:53:08.731532 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 15554
I20260929 01:53:08.748138 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 15699
W20260929 01:53:08.751075 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 15699: Not authorized: Client connection negotiation failed: client connection to 127.0.71.67:46651: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:53:08.751231 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 15699
I20260929 01:53:08.767621 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 15837
W20260929 01:53:08.769981 16235 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:08.768590 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:39322 (local address 127.0.71.126:33145)
0929 01:53:08.768792 (+ 202us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:08.768801 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:08.768826 (+ 25us) server_negotiation.cc:408] Connection header received
0929 01:53:08.768884 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:53:08.768888 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:53:08.768948 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:53:08.769073 (+ 125us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:53:08.769761 (+ 688us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39322: BlockingRecv error: recv got EOF from 127.0.0.1:39322 (error 108)
Metrics: {"server-negotiator.queue_time_us":68}
W20260929 01:53:08.769963 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 15837: Not authorized: Client connection negotiation failed: client connection to 127.0.71.126:33145: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:53:08.770149 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 15837
I20260929 01:53:08.790477 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 15206
I20260929 01:53:08.791656 15267 generic_service.cc:196] Checking for leaks (request via RPC)
W20260929 01:53:08.858549 15298 consensus_peers.cc:597] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 -> Peer 4d0302fb3ffc48bca2138a0120f7c472 (127.0.71.126:33145): Couldn't send request to peer 4d0302fb3ffc48bca2138a0120f7c472. Status: Network error: Client connection negotiation failed: client connection to 127.0.71.126:33145: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 01:53:08.902058 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 15206
I20260929 01:53:08.922573 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 15283
I20260929 01:53:08.923697 15344 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:53:09.039819 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 15283
Sep 29 01:53:09 dist-test-slave-vz23 krb5kdc[16249](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:53:09 dist-test-slave-vz23 krb5kdc[16249](info): set up 2 sockets
Sep 29 01:53:09 dist-test-slave-vz23 krb5kdc[16249](info): commencing operation
krb5kdc: starting...
W20260929 01:53:11.070948 285 mini_kdc.cc:121] Time spent starting KDC: real 2.007s user 0.003s sys 0.011s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-admin@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-user@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "joe-interloper@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 01:53:11 dist-test-slave-vz23 krb5kdc[16249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.126@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.126@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:11.255810 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:33145
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:53:11.589112 16263 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:11.589462 16263 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:11.589628 16263 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:11.597868 16263 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:53:11.597968 16263 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:11.598078 16263 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:53:11.598157 16263 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:53:11.608695 16263 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:33145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:11.610321 16263 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:11.611656 16263 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:11.620007 16269 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:11.619949 16268 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:11.620575 16271 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:11.621057 16263 server_base.cc:1061] running on GCE node
I20260929 01:53:11.621810 16263 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:53:11.621992 16263 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:53:11.622095 16263 hybrid_clock.cc:648] HybridClock initialized: now 1790646791622073 us; error 0 us; skew 500 ppm
I20260929 01:53:11.622422 16263 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:11 dist-test-slave-vz23 krb5kdc[16249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646791, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:11.626148 16263 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:53:11.627909 16263 webserver.cc:533] Webserver started at http://127.0.71.126:39641/ using document root <none> and password file <none>
I20260929 01:53:11.628683 16263 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:11.628782 16263 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:11.633896 16263 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.001s sys 0.003s
I20260929 01:53:11.636704 16277 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:11.638136 16263 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:53:11.638314 16263 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "4d0302fb3ffc48bca2138a0120f7c472"
format_stamp: "Formatted at 2026-09-29 01:53:02 on dist-test-slave-vz23"
I20260929 01:53:11.639070 16263 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:11.681115 16263 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:11.692729 16263 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:11.693046 16263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:11.717691 16328 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:33145 every 8 connection(s)
I20260929 01:53:11.717725 16263 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:33145
I20260929 01:53:11.720029 16263 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:53:11.724071 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 16263
I20260929 01:53:11.724256 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:53:11.728717 16329 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:53:11.733349 16329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: Bootstrap starting.
WARNING: no policy specified for kudu/127.0.71.125@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.125@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:53:11.752750 16329 log.cc:826] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: Log is configured to *not* fsync() on all Append() calls
I20260929 01:53:11.761185 16329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:53:11.761735 16329 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: Bootstrap complete.
I20260929 01:53:11.769846 16329 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } }
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.125 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.125 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:11.770763 16329 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0302fb3ffc48bca2138a0120f7c472, State: Initialized, Role: FOLLOWER
I20260929 01:53:11.771709 16329 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } }
I20260929 01:53:11.772653 16334 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:11.772919 16334 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:11.774080 16329 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:53:11.792922 16346 catalog_manager.cc:1308] Loaded cluster ID: 71984c3cb0d0405c972b708a17c5de2e
I20260929 01:53:11.793184 16346 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: loading cluster ID for follower catalog manager: success
WARNING: no policy specified for HTTP/127.0.71.125@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.125@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:53:11.798277 16346 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: acquiring CA information for follower catalog manager: success
I20260929 01:53:11.801128 16346 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:53:11.816069 16263 master_runner.cc:187] Error getting master registration for 127.0.71.125:34637: Network error: Client connection negotiation failed: client connection to 127.0.71.125:34637: connect: Connection refused (error 111),
I20260929 01:53:11.821342 16263 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.125 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.125 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:11.827759 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:34637
--webserver_interface=127.0.71.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:53:12.164800 16351 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:12.165134 16351 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:12.165334 16351 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:12.173884 16351 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:53:12.174000 16351 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:12.174101 16351 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:53:12.174167 16351 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:53:12.185436 16351 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:34637
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:12.187003 16351 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:12.188385 16351 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:12.196229 16356 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:12.196195 16357 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:12.197160 16359 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:12.197925 16351 server_base.cc:1061] running on GCE node
I20260929 01:53:12.198660 16351 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:53:12.198839 16351 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:53:12.198937 16351 hybrid_clock.cc:648] HybridClock initialized: now 1790646792198901 us; error 0 us; skew 500 ppm
I20260929 01:53:12.199280 16351 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:12 dist-test-slave-vz23 krb5kdc[16249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:12.202842 16351 init.cc:377] Logged in from keytab as kudu/127.0.71.125@KRBTEST.COM (short username kudu)
I20260929 01:53:12.204807 16351 webserver.cc:533] Webserver started at http://127.0.71.125:33333/ using document root <none> and password file <none>
I20260929 01:53:12.205534 16351 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:12.205677 16351 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:12.210954 16351 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.002s
I20260929 01:53:12.213896 16365 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:12.215227 16351 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:53:12.215400 16351 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "201815f40c9a4697b4bc751d4fed537d"
format_stamp: "Formatted at 2026-09-29 01:53:03 on dist-test-slave-vz23"
I20260929 01:53:12.216130 16351 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:12.230839 16351 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:12.234663 16351 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:12.234929 16351 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:12.255743 16351 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:34637
I20260929 01:53:12.255764 16416 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:34637 every 8 connection(s)
I20260929 01:53:12.257433 16351 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:53:12.258898 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 16351
I20260929 01:53:12.259097 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:53:12.264739 16417 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:53:12.270033 16417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d: Bootstrap starting.
WARNING: no policy specified for kudu/127.0.71.124@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.124@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:53:12.291275 16417 log.cc:826] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:53:12.300401 16417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:53:12.301050 16417 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d: Bootstrap complete.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.124 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.124 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:12.309407 16417 raft_consensus.cc:359] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } }
I20260929 01:53:12.310099 16417 raft_consensus.cc:740] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 201815f40c9a4697b4bc751d4fed537d, State: Initialized, Role: FOLLOWER
I20260929 01:53:12.310935 16417 consensus_queue.cc:260] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } }
I20260929 01:53:12.311820 16422 sys_catalog.cc:455] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:12.312130 16422 sys_catalog.cc:458] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:12.312849 16417 sys_catalog.cc:565] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:53:12.323197 16434 catalog_manager.cc:1308] Loaded cluster ID: 71984c3cb0d0405c972b708a17c5de2e
I20260929 01:53:12.323401 16434 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d: loading cluster ID for follower catalog manager: success
WARNING: no policy specified for HTTP/127.0.71.124@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.124@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:53:12.328547 16434 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d: acquiring CA information for follower catalog manager: success
I20260929 01:53:12.331593 16434 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.124 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.124 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:12.362193 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:38615
--webserver_interface=127.0.71.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:12.363611 16351 master_runner.cc:187] Error getting master registration for 127.0.71.124:38615: Network error: Client connection negotiation failed: client connection to 127.0.71.124:38615: connect: Connection refused (error 111),
I20260929 01:53:12.370012 16351 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:53:12.725081 16440 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:12.725546 16440 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:12.725715 16440 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:12.734151 16440 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:53:12.734297 16440 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:12.734414 16440 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:53:12.734503 16440 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:53:12.745350 16440 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:38615
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:12.747067 16440 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:12.748430 16440 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:12.756718 16445 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:12.758910 16446 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:12.759249 16448 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:12.759595 16440 server_base.cc:1061] running on GCE node
I20260929 01:53:12.760509 16440 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:53:12.760700 16440 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:53:12.760811 16440 hybrid_clock.cc:648] HybridClock initialized: now 1790646792760783 us; error 0 us; skew 500 ppm
I20260929 01:53:12.761231 16440 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:12 dist-test-slave-vz23 krb5kdc[16249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646792, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:12.765830 16440 init.cc:377] Logged in from keytab as kudu/127.0.71.124@KRBTEST.COM (short username kudu)
I20260929 01:53:12.768210 16440 webserver.cc:533] Webserver started at http://127.0.71.124:39523/ using document root <none> and password file <none>
I20260929 01:53:12.769076 16440 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:12.769235 16440 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:12.774982 16440 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:53:12.778301 16454 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:12.779806 16440 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:53:12.779986 16440 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "d7f07b2f531747c4a9985e51eba98357"
format_stamp: "Formatted at 2026-09-29 01:53:03 on dist-test-slave-vz23"
I20260929 01:53:12.780805 16440 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:12.809115 16440 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:12.813388 16440 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:12.813725 16440 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:12.834518 16440 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:38615
I20260929 01:53:12.834539 16505 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:38615 every 8 connection(s)
I20260929 01:53:12.836112 16440 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:53:12.839069 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 16440
I20260929 01:53:12.839267 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:53:12.844887 16506 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:53:12.849449 16506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357: Bootstrap starting.
Sep 29 01:53:12 dist-test-slave-vz23 krb5kdc[16249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:53:12 dist-test-slave-vz23 krb5kdc[16249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:53:12.867475 16506 log.cc:826] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357: Log is configured to *not* fsync() on all Append() calls
I20260929 01:53:12.878706 16506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:53:12.879271 16506 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357: Bootstrap complete.
Sep 29 01:53:12 dist-test-slave-vz23 krb5kdc[16249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:53:12.886559 16506 raft_consensus.cc:359] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } }
I20260929 01:53:12.887485 16506 raft_consensus.cc:740] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d7f07b2f531747c4a9985e51eba98357, State: Initialized, Role: FOLLOWER
I20260929 01:53:12.888262 16506 consensus_queue.cc:260] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } }
I20260929 01:53:12.889323 16511 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:12.889667 16511 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:12.890478 16506 sys_catalog.cc:565] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:53:12.901968 16522 catalog_manager.cc:1308] Loaded cluster ID: 71984c3cb0d0405c972b708a17c5de2e
I20260929 01:53:12.902092 16522 catalog_manager.cc:1601] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357: loading cluster ID for follower catalog manager: success
I20260929 01:53:12.906193 16522 catalog_manager.cc:1623] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357: acquiring CA information for follower catalog manager: success
I20260929 01:53:12.909032 16522 catalog_manager.cc:1651] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:53:12.937727 16440 master_runner.cc:264] No leader master found from master d7f07b2f531747c4a9985e51eba98357
I20260929 01:53:12.937849 16440 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:53:13.055291 16525 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:53:13.055611 16525 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } }
I20260929 01:53:13.057700 16525 leader_election.cc:290] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 201815f40c9a4697b4bc751d4fed537d (127.0.71.125:34637), d7f07b2f531747c4a9985e51eba98357 (127.0.71.124:38615)
I20260929 01:53:13.066208 16392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d0302fb3ffc48bca2138a0120f7c472" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "201815f40c9a4697b4bc751d4fed537d" is_pre_election: true
I20260929 01:53:13.066430 16481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d0302fb3ffc48bca2138a0120f7c472" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7f07b2f531747c4a9985e51eba98357" is_pre_election: true
I20260929 01:53:13.066752 16392 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d0302fb3ffc48bca2138a0120f7c472 in term 1.
I20260929 01:53:13.067032 16481 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4d0302fb3ffc48bca2138a0120f7c472 in term 1.
I20260929 01:53:13.067742 16281 leader_election.cc:304] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d0302fb3ffc48bca2138a0120f7c472, d7f07b2f531747c4a9985e51eba98357; no voters:
I20260929 01:53:13.068578 16525 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 01:53:13.068737 16525 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:53:13.068827 16525 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:53:13.074275 16525 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } }
I20260929 01:53:13.075050 16525 leader_election.cc:290] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [CANDIDATE]: Term 2 election: Requested vote from peers 201815f40c9a4697b4bc751d4fed537d (127.0.71.125:34637), d7f07b2f531747c4a9985e51eba98357 (127.0.71.124:38615)
I20260929 01:53:13.075529 16392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d0302fb3ffc48bca2138a0120f7c472" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "201815f40c9a4697b4bc751d4fed537d"
I20260929 01:53:13.075645 16481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4d0302fb3ffc48bca2138a0120f7c472" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d7f07b2f531747c4a9985e51eba98357"
I20260929 01:53:13.075743 16392 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:53:13.075961 16481 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:53:13.080032 16481 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d0302fb3ffc48bca2138a0120f7c472 in term 2.
I20260929 01:53:13.080554 16281 leader_election.cc:304] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 4d0302fb3ffc48bca2138a0120f7c472, d7f07b2f531747c4a9985e51eba98357; no voters:
I20260929 01:53:13.080915 16525 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:53:13.080991 16392 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4d0302fb3ffc48bca2138a0120f7c472 in term 2.
I20260929 01:53:13.081451 16525 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [term 2 LEADER]: Becoming Leader. State: Replica: 4d0302fb3ffc48bca2138a0120f7c472, State: Running, Role: LEADER
I20260929 01:53:13.082010 16525 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } }
I20260929 01:53:13.085049 16528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d0302fb3ffc48bca2138a0120f7c472. Latest consensus state: current_term: 2 leader_uuid: "4d0302fb3ffc48bca2138a0120f7c472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:13.085364 16528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [sys.catalog]: This master's current role is: LEADER
I20260929 01:53:13.087426 16530 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:53:13.089442 16530 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:53:13.090237 16530 catalog_manager.cc:1308] Loaded cluster ID: 71984c3cb0d0405c972b708a17c5de2e
I20260929 01:53:13.090340 16530 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:53:13.091485 16530 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:53:13.092331 16530 catalog_manager.cc:6230] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: Loaded TSK: 0
I20260929 01:53:13.093217 16530 catalog_manager.cc:1563] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.65@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.65@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:13.202071 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:13.496279 16481 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [term 2 FOLLOWER]: Refusing update from remote peer 4d0302fb3ffc48bca2138a0120f7c472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 01:53:13.497422 16528 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:53:13.501451 16536 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d0302fb3ffc48bca2138a0120f7c472. Latest consensus state: current_term: 2 leader_uuid: "4d0302fb3ffc48bca2138a0120f7c472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:13.501693 16536 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:13.503846 16536 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "4d0302fb3ffc48bca2138a0120f7c472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:13.504146 16536 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:13.506368 16392 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [term 2 FOLLOWER]: Refusing update from remote peer 4d0302fb3ffc48bca2138a0120f7c472: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 01:53:13.507254 16528 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:53:13.509090 16540 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "4d0302fb3ffc48bca2138a0120f7c472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:13.509347 16540 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [sys.catalog]: This master's current role is: LEADER
I20260929 01:53:13.511637 16544 sys_catalog.cc:455] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "4d0302fb3ffc48bca2138a0120f7c472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:13.512070 16544 sys_catalog.cc:458] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:13.512385 16541 sys_catalog.cc:455] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4d0302fb3ffc48bca2138a0120f7c472. Latest consensus state: current_term: 2 leader_uuid: "4d0302fb3ffc48bca2138a0120f7c472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:13.512701 16541 sys_catalog.cc:458] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:53:13.545420 16535 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:13.545755 16535 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:13.545934 16535 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:13.554383 16535 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:13.554629 16535 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:53:13.565701 16535 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:13.567417 16535 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:13.568876 16535 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:13.578869 16550 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:13.578929 16549 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:13.579399 16552 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:13.580274 16535 server_base.cc:1061] running on GCE node
I20260929 01:53:13.580963 16535 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:53:13.581171 16535 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:53:13.581277 16535 hybrid_clock.cc:648] HybridClock initialized: now 1790646793581255 us; error 0 us; skew 500 ppm
I20260929 01:53:13.581615 16535 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:13 dist-test-slave-vz23 krb5kdc[16249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:13.585654 16535 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:53:13.587457 16535 webserver.cc:533] Webserver started at http://127.0.71.65:44931/ using document root <none> and password file <none>
I20260929 01:53:13.588214 16535 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:13.588358 16535 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:13.594326 16535 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:53:13.597179 16558 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:13.598505 16535 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:53:13.598690 16535 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "25071766fd4a4a0e8c96d4a1df800197"
format_stamp: "Formatted at 2026-09-29 01:53:04 on dist-test-slave-vz23"
I20260929 01:53:13.599400 16535 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:13.618605 16535 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:13.623524 16535 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:13.623802 16535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:13.624900 16535 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:13.626412 16535 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:13.626544 16535 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:13.626674 16535 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:13.626761 16535 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:13.662470 16535 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:41741
I20260929 01:53:13.662559 16670 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:41741 every 8 connection(s)
I20260929 01:53:13.664012 16535 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:53:13.672647 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 16535
I20260929 01:53:13.672936 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
Sep 29 01:53:13 dist-test-slave-vz23 krb5kdc[16249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:53:13 dist-test-slave-vz23 krb5kdc[16249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:53:13.702468 16671 heartbeater.cc:344] Connected to a master server at 127.0.71.124:38615
I20260929 01:53:13.702875 16671 heartbeater.cc:461] Registering TS with master...
Sep 29 01:53:13 dist-test-slave-vz23 krb5kdc[16249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646793, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:53:13.703977 16671 heartbeater.cc:507] Master 127.0.71.124:38615 requested a full tablet report, sending...
I20260929 01:53:13.707546 16471 ts_manager.cc:194] Registered new tserver with Master: 25071766fd4a4a0e8c96d4a1df800197 (127.0.71.65:41741)
WARNING: no policy specified for kudu/127.0.71.66@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.66@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:53:13.710858 16672 heartbeater.cc:344] Connected to a master server at 127.0.71.126:33145
I20260929 01:53:13.711004 16672 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:13.711501 16672 heartbeater.cc:507] Master 127.0.71.126:33145 requested a full tablet report, sending...
I20260929 01:53:13.713011 16673 heartbeater.cc:344] Connected to a master server at 127.0.71.125:34637
I20260929 01:53:13.713116 16294 ts_manager.cc:194] Registered new tserver with Master: 25071766fd4a4a0e8c96d4a1df800197 (127.0.71.65:41741)
I20260929 01:53:13.713162 16673 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:13.713616 16673 heartbeater.cc:507] Master 127.0.71.125:34637 requested a full tablet report, sending...
I20260929 01:53:13.715194 16382 ts_manager.cc:194] Registered new tserver with Master: 25071766fd4a4a0e8c96d4a1df800197 (127.0.71.65:41741)
I20260929 01:53:13.715490 16294 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:50873
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.66 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.66 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.66@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.66@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.66 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.66 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:13.779091 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.66:0
--local_ip_for_outbound_sockets=127.0.71.66
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:14.028666 16686 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "4d0302fb3ffc48bca2138a0120f7c472" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:14.029188 16686 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [sys.catalog]: This master's current role is: LEADER
W20260929 01:53:14.109243 16685 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:14.109560 16685 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:14.109649 16685 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:14.118108 16685 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:14.118304 16685 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.66
I20260929 01:53:14.128945 16685 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.71.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:14.130517 16685 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:14.131829 16685 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:14.140694 16694 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:14.141144 16695 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:14.142083 16697 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:14.142951 16685 server_base.cc:1061] running on GCE node
I20260929 01:53:14.143558 16685 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:53:14.143759 16685 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:53:14.143864 16685 hybrid_clock.cc:648] HybridClock initialized: now 1790646794143826 us; error 0 us; skew 500 ppm
I20260929 01:53:14.144155 16685 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:14 dist-test-slave-vz23 krb5kdc[16249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:14.147790 16685 init.cc:377] Logged in from keytab as kudu/127.0.71.66@KRBTEST.COM (short username kudu)
I20260929 01:53:14.149477 16685 webserver.cc:533] Webserver started at http://127.0.71.66:40641/ using document root <none> and password file <none>
I20260929 01:53:14.150174 16685 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:14.150388 16685 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:14.155848 16685 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:53:14.158664 16703 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:14.160027 16685 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:53:14.160192 16685 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/wal
uuid: "d3ab5a94294f4deab6f57b2fa18d9728"
format_stamp: "Formatted at 2026-09-29 01:53:05 on dist-test-slave-vz23"
I20260929 01:53:14.160945 16685 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:14.178020 16685 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:14.181905 16685 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:14.182118 16685 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:14.183120 16685 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:14.184599 16685 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:14.184684 16685 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:14.184762 16685 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:14.184818 16685 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:14.219909 16685 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.66:34229
I20260929 01:53:14.219992 16815 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.66:34229 every 8 connection(s)
I20260929 01:53:14.221608 16685 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
I20260929 01:53:14.230365 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 16685
I20260929 01:53:14.230612 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-1/wal/instance
Sep 29 01:53:14 dist-test-slave-vz23 krb5kdc[16249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:53:14 dist-test-slave-vz23 krb5kdc[16249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:53:14 dist-test-slave-vz23 krb5kdc[16249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:53:14.269791 16817 heartbeater.cc:344] Connected to a master server at 127.0.71.126:33145
I20260929 01:53:14.269831 16818 heartbeater.cc:344] Connected to a master server at 127.0.71.125:34637
I20260929 01:53:14.270203 16818 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:14.270210 16817 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:14.270659 16816 heartbeater.cc:344] Connected to a master server at 127.0.71.124:38615
I20260929 01:53:14.270787 16816 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:14.271492 16817 heartbeater.cc:507] Master 127.0.71.126:33145 requested a full tablet report, sending...
I20260929 01:53:14.271696 16818 heartbeater.cc:507] Master 127.0.71.125:34637 requested a full tablet report, sending...
I20260929 01:53:14.271521 16816 heartbeater.cc:507] Master 127.0.71.124:38615 requested a full tablet report, sending...
I20260929 01:53:14.273273 16294 ts_manager.cc:194] Registered new tserver with Master: d3ab5a94294f4deab6f57b2fa18d9728 (127.0.71.66:34229)
I20260929 01:53:14.273335 16382 ts_manager.cc:194] Registered new tserver with Master: d3ab5a94294f4deab6f57b2fa18d9728 (127.0.71.66:34229)
I20260929 01:53:14.273432 16471 ts_manager.cc:194] Registered new tserver with Master: d3ab5a94294f4deab6f57b2fa18d9728 (127.0.71.66:34229)
I20260929 01:53:14.274384 16294 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.66@KRBTEST.COM'} at 127.0.71.66:39475
WARNING: no policy specified for kudu/127.0.71.67@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.67@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.67 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.67 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.67@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.67@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.67 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.67 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:14.348562 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.67:0
--local_ip_for_outbound_sockets=127.0.71.67
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:53:14.691294 16830 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:14.691707 16830 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:14.691874 16830 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:14.700304 16830 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:14.700611 16830 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.67
I20260929 01:53:14.711602 16830 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.71.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:14.713231 16830 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:14.714622 16830 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260929 01:53:14.718336 16672 heartbeater.cc:499] Master 127.0.71.126:33145 was elected leader, sending a full tablet report...
W20260929 01:53:14.726661 16835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:14.726701 16836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:14.728194 16838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:14.728773 16830 server_base.cc:1061] running on GCE node
I20260929 01:53:14.729444 16830 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:53:14.729580 16830 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:53:14.729686 16830 hybrid_clock.cc:648] HybridClock initialized: now 1790646794729657 us; error 0 us; skew 500 ppm
I20260929 01:53:14.730018 16830 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:14 dist-test-slave-vz23 krb5kdc[16249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:14.733526 16830 init.cc:377] Logged in from keytab as kudu/127.0.71.67@KRBTEST.COM (short username kudu)
I20260929 01:53:14.735057 16830 webserver.cc:533] Webserver started at http://127.0.71.67:41421/ using document root <none> and password file <none>
I20260929 01:53:14.735718 16830 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:14.735823 16830 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:14.741505 16830 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:53:14.744379 16844 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:14.745862 16830 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:53:14.745990 16830 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/wal
uuid: "4a4de4f4a1c4419998114ad18ced1014"
format_stamp: "Formatted at 2026-09-29 01:53:06 on dist-test-slave-vz23"
I20260929 01:53:14.746836 16830 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:14.783349 16830 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:14.787245 16830 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:14.787456 16830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:14.788341 16830 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:14.789932 16830 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:14.790024 16830 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:14.790102 16830 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:14.790158 16830 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:14.824311 16830 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.67:40001
I20260929 01:53:14.824411 16956 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.67:40001 every 8 connection(s)
I20260929 01:53:14.825831 16830 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
I20260929 01:53:14.830771 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 16830
I20260929 01:53:14.830991 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-2/wal/instance
Sep 29 01:53:14 dist-test-slave-vz23 krb5kdc[16249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:53:14 dist-test-slave-vz23 krb5kdc[16249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:53:14 dist-test-slave-vz23 krb5kdc[16249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646794, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:53:14.859925 16959 heartbeater.cc:344] Connected to a master server at 127.0.71.125:34637
I20260929 01:53:14.860303 16959 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:14.861191 16959 heartbeater.cc:507] Master 127.0.71.125:34637 requested a full tablet report, sending...
I20260929 01:53:14.862938 16382 ts_manager.cc:194] Registered new tserver with Master: 4a4de4f4a1c4419998114ad18ced1014 (127.0.71.67:40001)
I20260929 01:53:14.864197 16958 heartbeater.cc:344] Connected to a master server at 127.0.71.126:33145
I20260929 01:53:14.864348 16958 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:14.864954 16958 heartbeater.cc:507] Master 127.0.71.126:33145 requested a full tablet report, sending...
I20260929 01:53:14.866088 16294 ts_manager.cc:194] Registered new tserver with Master: 4a4de4f4a1c4419998114ad18ced1014 (127.0.71.67:40001)
I20260929 01:53:14.866856 16957 heartbeater.cc:344] Connected to a master server at 127.0.71.124:38615
I20260929 01:53:14.866988 16957 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:14.867030 16294 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.67@KRBTEST.COM'} at 127.0.71.67:58623
I20260929 01:53:14.867472 16957 heartbeater.cc:507] Master 127.0.71.124:38615 requested a full tablet report, sending...
I20260929 01:53:14.868569 16471 ts_manager.cc:194] Registered new tserver with Master: 4a4de4f4a1c4419998114ad18ced1014 (127.0.71.67:40001)
I20260929 01:53:14.872730 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:53:14.879669 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:53:14.879832 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 16263
I20260929 01:53:14.880694 16324 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:53:14.990525 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 16263
I20260929 01:53:15.019869 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:33145
--webserver_interface=127.0.71.126
--webserver_port=39641
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:15.277040 16817 heartbeater.cc:499] Master 127.0.71.126:33145 was elected leader, sending a full tablet report...
W20260929 01:53:15.278452 16817 heartbeater.cc:646] Failed to heartbeat to 127.0.71.126:33145 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.71.126:33145: connect: Connection refused (error 111)
W20260929 01:53:15.359365 16968 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:15.359738 16968 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:15.359898 16968 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:15.370759 16968 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:53:15.370946 16968 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:15.371057 16968 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:53:15.371135 16968 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:53:15.383440 16968 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:33145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=39641
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:15.385501 16968 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:15.387354 16968 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:15.396407 16974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:15.396409 16975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:15.397128 16977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:15.398097 16968 server_base.cc:1061] running on GCE node
I20260929 01:53:15.399014 16968 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:53:15.399217 16968 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:53:15.399408 16968 hybrid_clock.cc:648] HybridClock initialized: now 1790646795399379 us; error 0 us; skew 500 ppm
I20260929 01:53:15.399798 16968 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:15 dist-test-slave-vz23 krb5kdc[16249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:15.404068 16968 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:53:15.406430 16968 webserver.cc:533] Webserver started at http://127.0.71.126:39641/ using document root <none> and password file <none>
I20260929 01:53:15.407228 16968 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:15.407399 16968 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:15.413586 16968 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 01:53:15.416757 16983 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:15.418166 16968 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:53:15.418371 16968 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "4d0302fb3ffc48bca2138a0120f7c472"
format_stamp: "Formatted at 2026-09-29 01:53:02 on dist-test-slave-vz23"
I20260929 01:53:15.419100 16968 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:15.439498 16968 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:15.443791 16968 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:15.444094 16968 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:15.465150 16968 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:33145
I20260929 01:53:15.465197 17034 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:33145 every 8 connection(s)
I20260929 01:53:15.466831 16968 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:53:15.473776 17035 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:53:15.475601 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 16968
I20260929 01:53:15.475921 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 16535
I20260929 01:53:15.477921 17035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: Bootstrap starting.
Sep 29 01:53:15 dist-test-slave-vz23 krb5kdc[16249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646791, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 01:53:15.494953 16666 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:53:15.495450 17035 log.cc:826] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: Log is configured to *not* fsync() on all Append() calls
I20260929 01:53:15.505146 17035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:53:15.505738 17035 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: Bootstrap complete.
I20260929 01:53:15.512601 17035 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } }
I20260929 01:53:15.513341 17035 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4d0302fb3ffc48bca2138a0120f7c472, State: Initialized, Role: FOLLOWER
I20260929 01:53:15.514225 17035 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } }
I20260929 01:53:15.515147 17041 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:15.515425 17041 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:15.516304 17035 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:53:15.540324 17052 catalog_manager.cc:1308] Loaded cluster ID: 71984c3cb0d0405c972b708a17c5de2e
I20260929 01:53:15.540572 17052 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: loading cluster ID for follower catalog manager: success
I20260929 01:53:15.544253 17052 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: acquiring CA information for follower catalog manager: success
I20260929 01:53:15.546818 17052 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Sep 29 01:53:15 dist-test-slave-vz23 krb5kdc[16249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646795, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:53:15.589905 16968 master_runner.cc:264] No leader master found from master 4d0302fb3ffc48bca2138a0120f7c472
I20260929 01:53:15.590076 16968 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:53:15.608199 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 16535
I20260929 01:53:15.626143 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:41741
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--webserver_port=44931
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:15.869663 16958 heartbeater.cc:499] Master 127.0.71.126:33145 was elected leader, sending a full tablet report...
I20260929 01:53:15.882012 17000 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "4a4de4f4a1c4419998114ad18ced1014" instance_seqno: 1790646794815916) as {username='kudu', principal='kudu/127.0.71.67@KRBTEST.COM'} at 127.0.71.67:37075; Asking this server to re-register.
I20260929 01:53:15.883121 16958 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:15.885046 17000 ts_manager.cc:194] Registered new tserver with Master: 4a4de4f4a1c4419998114ad18ced1014 (127.0.71.67:40001)
W20260929 01:53:15.971603 17058 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:15.971951 17058 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:15.972133 17058 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:15.980598 17058 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:15.980850 17058 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:53:15.991468 17058 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:41741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=44931
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:15.993614 17058 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:15.994948 17058 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:16.004917 17065 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:16.004905 17064 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:16.010080 17067 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:16.010578 17058 server_base.cc:1061] running on GCE node
I20260929 01:53:16.011169 17058 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:53:16.011358 17058 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:53:16.011431 17058 hybrid_clock.cc:648] HybridClock initialized: now 1790646796011413 us; error 0 us; skew 500 ppm
I20260929 01:53:16.011677 17058 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:16 dist-test-slave-vz23 krb5kdc[16249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:16.015379 17058 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:53:16.017241 17058 webserver.cc:533] Webserver started at http://127.0.71.65:44931/ using document root <none> and password file <none>
I20260929 01:53:16.018054 17058 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:16.018148 17058 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:16.024085 17058 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:53:16.027161 17073 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:16.028759 17058 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:53:16.028898 17058 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "25071766fd4a4a0e8c96d4a1df800197"
format_stamp: "Formatted at 2026-09-29 01:53:04 on dist-test-slave-vz23"
I20260929 01:53:16.029587 17058 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:16.035622 17076 raft_consensus.cc:493] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4d0302fb3ffc48bca2138a0120f7c472)
I20260929 01:53:16.035883 17076 raft_consensus.cc:515] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } }
I20260929 01:53:16.037923 17076 leader_election.cc:290] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4d0302fb3ffc48bca2138a0120f7c472 (127.0.71.126:33145), 201815f40c9a4697b4bc751d4fed537d (127.0.71.125:34637)
I20260929 01:53:16.038525 16392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7f07b2f531747c4a9985e51eba98357" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "201815f40c9a4697b4bc751d4fed537d" is_pre_election: true
I20260929 01:53:16.046088 17010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7f07b2f531747c4a9985e51eba98357" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d0302fb3ffc48bca2138a0120f7c472" is_pre_election: true
I20260929 01:53:16.046597 17010 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d7f07b2f531747c4a9985e51eba98357 in term 2.
I20260929 01:53:16.047200 16455 leader_election.cc:304] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d0302fb3ffc48bca2138a0120f7c472, d7f07b2f531747c4a9985e51eba98357; no voters: 201815f40c9a4697b4bc751d4fed537d
I20260929 01:53:16.047616 17076 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 01:53:16.047763 17076 raft_consensus.cc:493] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 4d0302fb3ffc48bca2138a0120f7c472)
I20260929 01:53:16.047869 17076 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:53:16.050900 17076 raft_consensus.cc:515] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } }
I20260929 01:53:16.051690 17076 leader_election.cc:290] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [CANDIDATE]: Term 3 election: Requested vote from peers 4d0302fb3ffc48bca2138a0120f7c472 (127.0.71.126:33145), 201815f40c9a4697b4bc751d4fed537d (127.0.71.125:34637)
I20260929 01:53:16.052239 17010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7f07b2f531747c4a9985e51eba98357" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d0302fb3ffc48bca2138a0120f7c472"
I20260929 01:53:16.052232 16392 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d7f07b2f531747c4a9985e51eba98357" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "201815f40c9a4697b4bc751d4fed537d"
I20260929 01:53:16.052488 17010 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:53:16.053247 17058 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:16.056740 17010 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d7f07b2f531747c4a9985e51eba98357 in term 3.
I20260929 01:53:16.057359 16455 leader_election.cc:304] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 4d0302fb3ffc48bca2138a0120f7c472, d7f07b2f531747c4a9985e51eba98357; no voters: 201815f40c9a4697b4bc751d4fed537d
I20260929 01:53:16.057758 17076 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 01:53:16.058202 17058 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:16.058257 17076 raft_consensus.cc:697] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [term 3 LEADER]: Becoming Leader. State: Replica: d7f07b2f531747c4a9985e51eba98357, State: Running, Role: LEADER
I20260929 01:53:16.058542 17058 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:16.058724 17076 consensus_queue.cc:237] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } }
I20260929 01:53:16.059877 17058 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:16.060683 17080 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7f07b2f531747c4a9985e51eba98357. Latest consensus state: current_term: 3 leader_uuid: "d7f07b2f531747c4a9985e51eba98357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:16.061028 17080 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [sys.catalog]: This master's current role is: LEADER
I20260929 01:53:16.061841 17083 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:53:16.062212 17058 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:16.062348 17058 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:16.062490 17058 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:16.062609 17058 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:16.064194 17083 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:53:16.065073 17083 catalog_manager.cc:1308] Loaded cluster ID: 71984c3cb0d0405c972b708a17c5de2e
I20260929 01:53:16.065179 17083 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:53:16.066349 17083 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:53:16.067298 17083 catalog_manager.cc:6230] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357: Loaded TSK: 0
I20260929 01:53:16.068436 17083 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:53:16.100744 17058 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:41741
I20260929 01:53:16.100786 17190 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:41741 every 8 connection(s)
I20260929 01:53:16.102372 17058 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:53:16.110775 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 17058
Sep 29 01:53:16 dist-test-slave-vz23 krb5kdc[16249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:53:16 dist-test-slave-vz23 krb5kdc[16249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:53:16 dist-test-slave-vz23 krb5kdc[16249](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:53:16.133107 17191 heartbeater.cc:344] Connected to a master server at 127.0.71.124:38615
I20260929 01:53:16.133536 17191 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:16.134721 17191 heartbeater.cc:507] Master 127.0.71.124:38615 requested a full tablet report, sending...
I20260929 01:53:16.135809 17192 heartbeater.cc:344] Connected to a master server at 127.0.71.126:33145
I20260929 01:53:16.135968 17192 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:16.136551 17192 heartbeater.cc:507] Master 127.0.71.126:33145 requested a full tablet report, sending...
I20260929 01:53:16.136858 16471 ts_manager.cc:194] Re-registered known tserver with Master: 25071766fd4a4a0e8c96d4a1df800197 (127.0.71.65:41741)
I20260929 01:53:16.137836 17000 ts_manager.cc:194] Registered new tserver with Master: 25071766fd4a4a0e8c96d4a1df800197 (127.0.71.65:41741)
I20260929 01:53:16.139104 16471 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:35111
I20260929 01:53:16.139879 17193 heartbeater.cc:344] Connected to a master server at 127.0.71.125:34637
I20260929 01:53:16.140031 17193 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:16.140586 17193 heartbeater.cc:507] Master 127.0.71.125:34637 requested a full tablet report, sending...
I20260929 01:53:16.141918 16382 ts_manager.cc:194] Re-registered known tserver with Master: 25071766fd4a4a0e8c96d4a1df800197 (127.0.71.65:41741)
I20260929 01:53:16.154955 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 17058
W20260929 01:53:16.156102 17203 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:16.152642 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:47782 (local address 127.0.71.65:41741)
0929 01:53:16.153365 (+ 723us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:16.153372 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:16.153400 (+ 28us) server_negotiation.cc:408] Connection header received
0929 01:53:16.153462 (+ 62us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:53:16.153469 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:53:16.153669 (+ 200us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:53:16.153844 (+ 175us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:53:16.154651 (+ 807us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47782: BlockingRecv error: recv got EOF from 127.0.0.1:47782 (error 108)
Metrics: {"server-negotiator.queue_time_us":599,"thread_start_us":421,"threads_started":1}
W20260929 01:53:16.157334 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 17058: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:41741: server requires authentication, but client does not have Kerberos credentials available
W20260929 01:53:16.157446 17203 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:16.155907 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:47798 (local address 127.0.71.65:41741)
0929 01:53:16.156387 (+ 480us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:16.156393 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:16.156413 (+ 20us) server_negotiation.cc:408] Connection header received
0929 01:53:16.156486 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:53:16.156490 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:53:16.156564 (+ 74us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:53:16.156663 (+ 99us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:53:16.157204 (+ 541us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:47798: BlockingRecv error: recv got EOF from 127.0.0.1:47798 (error 108)
Metrics: {"server-negotiator.queue_time_us":398,"thread_start_us":620,"threads_started":1}
I20260929 01:53:16.157526 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 17058
I20260929 01:53:16.174561 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:41741
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--webserver_port=44931
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:16.286511 16817 heartbeater.cc:344] Connected to a master server at 127.0.71.126:33145
I20260929 01:53:16.286760 16817 heartbeater.cc:499] Master 127.0.71.126:33145 was elected leader, sending a full tablet report...
I20260929 01:53:16.287766 17000 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d3ab5a94294f4deab6f57b2fa18d9728" instance_seqno: 1790646794211458) as {username='kudu', principal='kudu/127.0.71.66@KRBTEST.COM'} at 127.0.71.66:39005; Asking this server to re-register.
I20260929 01:53:16.288475 16817 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:16.289638 17000 ts_manager.cc:194] Registered new tserver with Master: d3ab5a94294f4deab6f57b2fa18d9728 (127.0.71.66:34229)
I20260929 01:53:16.350247 17207 raft_consensus.cc:493] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 4d0302fb3ffc48bca2138a0120f7c472)
I20260929 01:53:16.350520 17207 raft_consensus.cc:515] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } }
I20260929 01:53:16.352506 17207 leader_election.cc:290] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 4d0302fb3ffc48bca2138a0120f7c472 (127.0.71.126:33145), d7f07b2f531747c4a9985e51eba98357 (127.0.71.124:38615)
I20260929 01:53:16.360636 16481 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "201815f40c9a4697b4bc751d4fed537d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d7f07b2f531747c4a9985e51eba98357" is_pre_election: true
I20260929 01:53:16.360948 17010 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "201815f40c9a4697b4bc751d4fed537d" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "4d0302fb3ffc48bca2138a0120f7c472" is_pre_election: true
I20260929 01:53:16.361318 17010 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 201815f40c9a4697b4bc751d4fed537d in current term 3: Already voted for candidate d7f07b2f531747c4a9985e51eba98357 in this term.
I20260929 01:53:16.362077 16366 leader_election.cc:304] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 201815f40c9a4697b4bc751d4fed537d; no voters: 4d0302fb3ffc48bca2138a0120f7c472, d7f07b2f531747c4a9985e51eba98357
I20260929 01:53:16.362506 17207 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:53:16.364851 17207 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260929 01:53:16.519819 17205 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:16.520154 17205 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:16.520318 17205 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:53:16.520404 17205 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260929 01:53:16.526134 16392 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [term 3 FOLLOWER]: Refusing update from remote peer d7f07b2f531747c4a9985e51eba98357: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 01:53:16.526960 17080 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [LEADER]: Connected to new peer: Peer: permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:53:16.528908 17207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7f07b2f531747c4a9985e51eba98357. Latest consensus state: current_term: 3 leader_uuid: "d7f07b2f531747c4a9985e51eba98357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:16.529260 17207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:53:16.529654 17205 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:16.529932 17205 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:53:16.534727 17080 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "d7f07b2f531747c4a9985e51eba98357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:16.534964 17080 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [sys.catalog]: This master's current role is: LEADER
I20260929 01:53:16.536831 17010 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [term 3 FOLLOWER]: Refusing update from remote peer d7f07b2f531747c4a9985e51eba98357: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 01:53:16.537928 17076 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [LEADER]: Connected to new peer: Peer: permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:53:16.541347 17207 sys_catalog.cc:455] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "d7f07b2f531747c4a9985e51eba98357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:16.541604 17207 sys_catalog.cc:458] T 00000000000000000000000000000000 P 201815f40c9a4697b4bc751d4fed537d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:16.546690 17205 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:41741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=44931
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:16.548863 17205 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:16.550900 17205 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260929 01:53:16.551100 17219 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "d7f07b2f531747c4a9985e51eba98357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:16.551573 17219 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:16.551844 17216 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d7f07b2f531747c4a9985e51eba98357. Latest consensus state: current_term: 3 leader_uuid: "d7f07b2f531747c4a9985e51eba98357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:16.552181 17216 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4d0302fb3ffc48bca2138a0120f7c472 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:53:16.561410 17221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:16.561364 17222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:16.566004 17205 server_base.cc:1061] running on GCE node
W20260929 01:53:16.566313 17224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:16.566990 17205 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:53:16.567170 17205 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:53:16.567289 17205 hybrid_clock.cc:648] HybridClock initialized: now 1790646796567250 us; error 0 us; skew 500 ppm
I20260929 01:53:16.567592 17205 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:16 dist-test-slave-vz23 krb5kdc[16249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646796, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:16.571530 17205 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:53:16.573539 17205 webserver.cc:533] Webserver started at http://127.0.71.65:44931/ using document root <none> and password file <none>
I20260929 01:53:16.574286 17205 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:16.574406 17205 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:16.580024 17205 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 01:53:16.583042 17230 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:16.584645 17205 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 01:53:16.584816 17205 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "25071766fd4a4a0e8c96d4a1df800197"
format_stamp: "Formatted at 2026-09-29 01:53:04 on dist-test-slave-vz23"
I20260929 01:53:16.585613 17205 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:16.610708 17205 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:16.614868 17205 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:16.615164 17205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:16.616243 17205 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:16.618099 17205 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:16.618203 17205 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:16.618346 17205 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:16.618418 17205 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:53:16.618928 17205 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:53:16.619146 17205 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:53:16.619283 17205 fault_injection.cc:59] injecting fault for kudu (pid 17205): FLAGS_fault_before_start (process will exit)
I20260929 01:53:16.636225 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:41741
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--webserver_port=44931
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:16.950251 17076 sys_catalog.cc:455] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "d7f07b2f531747c4a9985e51eba98357" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "4d0302fb3ffc48bca2138a0120f7c472" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33145 } } peers { permanent_uuid: "201815f40c9a4697b4bc751d4fed537d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34637 } } peers { permanent_uuid: "d7f07b2f531747c4a9985e51eba98357" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 38615 } } }
I20260929 01:53:16.950563 17076 sys_catalog.cc:458] T 00000000000000000000000000000000 P d7f07b2f531747c4a9985e51eba98357 [sys.catalog]: This master's current role is: LEADER
W20260929 01:53:16.978878 17238 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:16.979174 17238 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:16.979298 17238 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:16.987758 17238 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:16.987949 17238 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:53:16.999181 17238 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:41741
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=44931
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:33145,127.0.71.125:34637,127.0.71.124:38615
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:17.000835 17238 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:17.002115 17238 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:17.012496 17244 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:17.013389 17245 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:17.014830 17247 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:17.018041 17238 server_base.cc:1061] running on GCE node
I20260929 01:53:17.018721 17238 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:53:17.018910 17238 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:53:17.019006 17238 hybrid_clock.cc:648] HybridClock initialized: now 1790646797018968 us; error 0 us; skew 500 ppm
I20260929 01:53:17.019413 17238 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:17 dist-test-slave-vz23 krb5kdc[16249](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646797, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:17.023334 17238 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:53:17.025215 17238 webserver.cc:533] Webserver started at http://127.0.71.65:44931/ using document root <none> and password file <none>
I20260929 01:53:17.025906 17238 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:17.026000 17238 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:17.031590 17238 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:53:17.034497 17253 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:17.035853 17238 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:53:17.036054 17238 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "25071766fd4a4a0e8c96d4a1df800197"
format_stamp: "Formatted at 2026-09-29 01:53:04 on dist-test-slave-vz23"
I20260929 01:53:17.036811 17238 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:17.052011 17238 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:17.062813 17238 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:17.063247 17238 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:17.064594 17238 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:17.066349 17238 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:17.066478 17238 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:17.066607 17238 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:17.066696 17238 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:17.103152 17238 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:41741
I20260929 01:53:17.103200 17365 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:41741 every 8 connection(s)
I20260929 01:53:17.104792 17238 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:53:17.111735 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 17238
I20260929 01:53:17.112109 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 17238
W20260929 01:53:17.131460 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 17238: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:41741: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:53:17.131666 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 17238
I20260929 01:53:17.151443 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 16685
W20260929 01:53:17.152136 17370 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:17.108198 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:41023 (local address 127.0.71.126:33145)
0929 01:53:17.124430 (+ 16232us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:17.124456 (+ 26us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:17.128524 (+ 4068us) server_negotiation.cc:408] Connection header received
0929 01:53:17.128611 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:53:17.128621 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:53:17.128728 (+ 107us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:53:17.128857 (+ 129us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:53:17.151189 (+ 22332us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:41023: BlockingRecv error: recv got EOF from 127.0.71.65:41023 (error 108)
Metrics: {"server-negotiator.queue_time_us":16068,"thread_start_us":20145,"threads_started":1}
W20260929 01:53:17.152140 17376 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:17.131535 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:53607 (local address 127.0.71.125:34637)
0929 01:53:17.136856 (+ 5321us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:17.136860 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:17.151135 (+ 14275us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:53607: BlockingRecv error: recv got EOF from 127.0.71.65:53607 (error 108)
Metrics: {"server-negotiator.queue_time_us":5232,"thread_start_us":5089,"threads_started":1}
W20260929 01:53:17.152209 17371 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:17.108179 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:49083 (local address 127.0.71.124:38615)
0929 01:53:17.131716 (+ 23537us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:17.131723 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:17.132292 (+ 569us) server_negotiation.cc:408] Connection header received
0929 01:53:17.132359 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:53:17.132368 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:53:17.132491 (+ 123us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:53:17.132596 (+ 105us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:53:17.151159 (+ 18563us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:49083: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":23398,"thread_start_us":24858,"threads_started":1}
W20260929 01:53:17.154726 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 16685: Not authorized: Client connection negotiation failed: client connection to 127.0.71.66:34229: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:53:17.154892 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 16685
I20260929 01:53:17.171226 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 16830
W20260929 01:53:17.174022 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 16830: Not authorized: Client connection negotiation failed: client connection to 127.0.71.67:40001: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:53:17.174166 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 16830
I20260929 01:53:17.189918 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 16968
W20260929 01:53:17.191951 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 16968: Not authorized: Client connection negotiation failed: client connection to 127.0.71.126:33145: server requires authentication, but client does not have Kerberos credentials available
W20260929 01:53:17.192090 17370 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:17.190825 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:42314 (local address 127.0.71.126:33145)
0929 01:53:17.190998 (+ 173us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:17.191002 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:17.191020 (+ 18us) server_negotiation.cc:408] Connection header received
0929 01:53:17.191077 (+ 57us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:53:17.191085 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:53:17.191145 (+ 60us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:53:17.191226 (+ 81us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:53:17.191837 (+ 611us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42314: BlockingRecv error: recv got EOF from 127.0.0.1:42314 (error 108)
Metrics: {"server-negotiator.queue_time_us":55}
I20260929 01:53:17.192147 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 16968
I20260929 01:53:17.212792 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 16351
I20260929 01:53:17.213963 16412 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:53:17.325834 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 16351
I20260929 01:53:17.346194 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 16440
I20260929 01:53:17.347491 16501 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:53:17.461571 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 16440
[ OK ] ExternalMiniClusterTest.TestBasicOperation/6 (17485 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/7
Loading random data
Initializing database '/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:53:17 dist-test-slave-vz23 krb5kdc[17383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:53:17 dist-test-slave-vz23 krb5kdc[17383](info): set up 2 sockets
Sep 29 01:53:17 dist-test-slave-vz23 krb5kdc[17383](info): commencing operation
krb5kdc: starting...
W20260929 01:53:19.637250 285 mini_kdc.cc:121] Time spent starting KDC: real 2.140s user 0.000s sys 0.020s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 01:53:19 dist-test-slave-vz23 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:53:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:53:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:19.908041 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:43705
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:33449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:53:20.244942 17399 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:20.245318 17399 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:20.245465 17399 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:20.253871 17399 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:53:20.254019 17399 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:20.254145 17399 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:53:20.254235 17399 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:53:20.265066 17399 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:43705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:20.266769 17399 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:20.268133 17399 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:20.276973 17404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:20.277400 17407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:20.277309 17405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:20.278126 17399 server_base.cc:1061] running on GCE node
I20260929 01:53:20.278910 17399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:20.280305 17399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:20.281549 17399 hybrid_clock.cc:648] HybridClock initialized: now 1790646800281516 us; error 47 us; skew 500 ppm
I20260929 01:53:20.281958 17399 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:20 dist-test-slave-vz23 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:20.285198 17399 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:53:20.286754 17399 webserver.cc:533] Webserver started at http://127.0.71.126:40491/ using document root <none> and password file <none>
I20260929 01:53:20.287477 17399 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:20.287624 17399 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:20.287972 17399 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:53:20.290421 17399 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "0419fca067cd4c31ad19f304709d4d94"
format_stamp: "Formatted at 2026-09-29 01:53:20 on dist-test-slave-vz23"
I20260929 01:53:20.291191 17399 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "0419fca067cd4c31ad19f304709d4d94"
format_stamp: "Formatted at 2026-09-29 01:53:20 on dist-test-slave-vz23"
I20260929 01:53:20.296674 17399 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260929 01:53:20.300338 17414 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:20.301812 17399 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:53:20.301991 17399 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "0419fca067cd4c31ad19f304709d4d94"
format_stamp: "Formatted at 2026-09-29 01:53:20 on dist-test-slave-vz23"
I20260929 01:53:20.302182 17399 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:20.335435 17399 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:20.339490 17399 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:20.339788 17399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:20.360590 17399 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:43705
I20260929 01:53:20.360643 17465 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:43705 every 8 connection(s)
I20260929 01:53:20.362572 17399 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:53:20.367139 17466 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:53:20.369345 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 17399
I20260929 01:53:20.369561 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:53:20.376165 17466 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.71.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.125@KRBTEST.COM" created.
Sep 29 01:53:20 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:53:20.405488 17466 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:53:20.406663 17418 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.125:40265: connect: Connection refused (error 111)
W20260929 01:53:20.408396 17466 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:40265: Network error: Client connection negotiation failed: client connection to 127.0.71.125:40265: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.125@KRBTEST.COM" created.
I20260929 01:53:20.457796 17466 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } attempt: 1
W20260929 01:53:20.460235 17466 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:40265: Network error: Client connection negotiation failed: client connection to 127.0.71.125:40265: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:20.472585 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:40265
--webserver_interface=127.0.71.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:33449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:20.528549 17466 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } attempt: 2
W20260929 01:53:20.530804 17466 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:40265: Network error: Client connection negotiation failed: client connection to 127.0.71.125:40265: connect: Connection refused (error 111)
I20260929 01:53:20.622109 17466 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } attempt: 3
W20260929 01:53:20.624397 17466 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:40265: Network error: Client connection negotiation failed: client connection to 127.0.71.125:40265: connect: Connection refused (error 111)
I20260929 01:53:20.767748 17466 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } attempt: 4
W20260929 01:53:20.770542 17466 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:40265: Network error: Client connection negotiation failed: client connection to 127.0.71.125:40265: connect: Connection refused (error 111)
W20260929 01:53:20.808493 17475 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:20.808801 17475 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:20.808943 17475 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:20.817425 17475 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:53:20.817576 17475 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:20.817975 17475 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:53:20.818140 17475 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:53:20.830559 17475 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:40265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:20.832296 17475 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:20.833755 17475 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:20.842216 17480 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:20.842191 17481 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:20.843199 17483 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:20.843874 17475 server_base.cc:1061] running on GCE node
I20260929 01:53:20.844815 17475 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:20.846290 17475 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:20.847513 17475 hybrid_clock.cc:648] HybridClock initialized: now 1790646800847470 us; error 33 us; skew 500 ppm
I20260929 01:53:20.847937 17475 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:20 dist-test-slave-vz23 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:20.851554 17475 init.cc:377] Logged in from keytab as kudu/127.0.71.125@KRBTEST.COM (short username kudu)
I20260929 01:53:20.853533 17475 webserver.cc:533] Webserver started at http://127.0.71.125:33293/ using document root <none> and password file <none>
I20260929 01:53:20.854295 17475 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:20.854435 17475 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:20.854789 17475 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:53:20.857491 17475 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/data/instance:
uuid: "222b3cc4d7c0489c8760afddb2484746"
format_stamp: "Formatted at 2026-09-29 01:53:20 on dist-test-slave-vz23"
I20260929 01:53:20.858317 17475 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/wal/instance:
uuid: "222b3cc4d7c0489c8760afddb2484746"
format_stamp: "Formatted at 2026-09-29 01:53:20 on dist-test-slave-vz23"
I20260929 01:53:20.864214 17475 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 01:53:20.868072 17490 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:20.869601 17475 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:53:20.869827 17475 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "222b3cc4d7c0489c8760afddb2484746"
format_stamp: "Formatted at 2026-09-29 01:53:20 on dist-test-slave-vz23"
I20260929 01:53:20.870033 17475 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:20.885576 17475 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:20.890038 17475 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:20.890345 17475 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:20.911478 17475 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:40265
I20260929 01:53:20.911517 17541 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:40265 every 8 connection(s)
I20260929 01:53:20.913141 17475 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:53:20.914372 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 17475
I20260929 01:53:20.914590 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:53:20.918829 17542 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:53:20.925724 17542 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.71.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.124@KRBTEST.COM" created.
Sep 29 01:53:20 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:53:20.948249 17542 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:53:20 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:53:20.966303 17542 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:53:20.967657 17492 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.124:34639: connect: Connection refused (error 111)
W20260929 01:53:20.969431 17542 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:34639: Network error: Client connection negotiation failed: client connection to 127.0.71.124:34639: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.124@KRBTEST.COM" created.
I20260929 01:53:21.018754 17542 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } attempt: 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:21.020174 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:34639
--webserver_interface=127.0.71.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:33449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:53:21.021010 17542 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:34639: Network error: Client connection negotiation failed: client connection to 127.0.71.124:34639: connect: Connection refused (error 111)
I20260929 01:53:21.069835 17466 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } attempt: 5
Sep 29 01:53:21 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:53:21.083653 17466 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:53:21.085850 17466 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:34639: Network error: Client connection negotiation failed: client connection to 127.0.71.124:34639: connect: Connection refused (error 111)
I20260929 01:53:21.089239 17542 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } attempt: 2
W20260929 01:53:21.091056 17542 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:34639: Network error: Client connection negotiation failed: client connection to 127.0.71.124:34639: connect: Connection refused (error 111)
I20260929 01:53:21.137074 17466 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } attempt: 1
W20260929 01:53:21.139153 17466 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:34639: Network error: Client connection negotiation failed: client connection to 127.0.71.124:34639: connect: Connection refused (error 111)
I20260929 01:53:21.182294 17542 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } attempt: 3
W20260929 01:53:21.184526 17542 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:34639: Network error: Client connection negotiation failed: client connection to 127.0.71.124:34639: connect: Connection refused (error 111)
I20260929 01:53:21.207401 17466 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } attempt: 2
W20260929 01:53:21.209431 17466 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:34639: Network error: Client connection negotiation failed: client connection to 127.0.71.124:34639: connect: Connection refused (error 111)
I20260929 01:53:21.315716 17466 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } attempt: 3
W20260929 01:53:21.317924 17466 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:34639: Network error: Client connection negotiation failed: client connection to 127.0.71.124:34639: connect: Connection refused (error 111)
I20260929 01:53:21.327769 17542 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } attempt: 4
W20260929 01:53:21.330148 17542 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:34639: Network error: Client connection negotiation failed: client connection to 127.0.71.124:34639: connect: Connection refused (error 111)
W20260929 01:53:21.372689 17552 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:21.373003 17552 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:21.373153 17552 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:21.381562 17552 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:53:21.381660 17552 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:21.381771 17552 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:53:21.381822 17552 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:53:21.394481 17552 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:34639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:21.396093 17552 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:21.397503 17552 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:21.405920 17558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:21.406271 17557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:21.407938 17552 server_base.cc:1061] running on GCE node
W20260929 01:53:21.408011 17560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:21.408952 17552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:21.410364 17552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:21.411561 17552 hybrid_clock.cc:648] HybridClock initialized: now 1790646801411517 us; error 37 us; skew 500 ppm
I20260929 01:53:21.411860 17552 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:21 dist-test-slave-vz23 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:21.415177 17552 init.cc:377] Logged in from keytab as kudu/127.0.71.124@KRBTEST.COM (short username kudu)
I20260929 01:53:21.416893 17552 webserver.cc:533] Webserver started at http://127.0.71.124:38299/ using document root <none> and password file <none>
I20260929 01:53:21.417567 17552 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:21.417660 17552 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:21.418001 17552 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:53:21.420392 17552 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/data/instance:
uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092"
format_stamp: "Formatted at 2026-09-29 01:53:21 on dist-test-slave-vz23"
I20260929 01:53:21.421236 17552 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/wal/instance:
uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092"
format_stamp: "Formatted at 2026-09-29 01:53:21 on dist-test-slave-vz23"
I20260929 01:53:21.426729 17552 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.003s
I20260929 01:53:21.430388 17567 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:21.431872 17552 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:53:21.432060 17552 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092"
format_stamp: "Formatted at 2026-09-29 01:53:21 on dist-test-slave-vz23"
I20260929 01:53:21.432271 17552 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:21.445621 17552 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:21.449548 17552 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:21.449826 17552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:21.470674 17552 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:34639
I20260929 01:53:21.470697 17618 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:34639 every 8 connection(s)
I20260929 01:53:21.472385 17552 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:53:21.476841 17619 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:53:21.482657 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 17552
I20260929 01:53:21.482882 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:53:21.482865 17619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:53:21 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:53:21.495177 17466 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } attempt: 4
Sep 29 01:53:21 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:53:21 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:53:21 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:53:21.511358 17619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:53:21.520394 17466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: Bootstrap starting.
Sep 29 01:53:21 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:53:21.524801 17466 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: Neither blocks nor log segments found. Creating new log.
I20260929 01:53:21.526253 17466 log.cc:826] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: Log is configured to *not* fsync() on all Append() calls
Sep 29 01:53:21 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:53:21.532090 17619 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:53:21.535679 17466 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: No bootstrap required, opened a new log
Sep 29 01:53:21 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646801, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:53:21.542116 17466 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } }
I20260929 01:53:21.542407 17466 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:53:21.542563 17466 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0419fca067cd4c31ad19f304709d4d94, State: Initialized, Role: FOLLOWER
I20260929 01:53:21.543394 17466 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } }
I20260929 01:53:21.544538 17629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:21.544826 17629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:21.545666 17466 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:53:21.552878 17619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092: Bootstrap starting.
W20260929 01:53:21.556900 17640 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260929 01:53:21.556942 17619 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092: Neither blocks nor log segments found. Creating new log.
W20260929 01:53:21.557149 17640 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:53:21.558185 17619 log.cc:826] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092: Log is configured to *not* fsync() on all Append() calls
I20260929 01:53:21.556486 17399 master_runner.cc:187] Error getting master registration for 127.0.71.125:40265: OK, instance_id { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" instance_seqno: 1790646800905698 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:53:21.565829 17399 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:53:21.566942 17619 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092: No bootstrap required, opened a new log
I20260929 01:53:21.572672 17619 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } }
I20260929 01:53:21.572929 17619 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:53:21.573066 17619 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c7ddf0bc0cb4650ae96cbe167b2e092, State: Initialized, Role: FOLLOWER
I20260929 01:53:21.573643 17619 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } }
I20260929 01:53:21.574571 17643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:21.574872 17643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:21.575809 17619 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:53:21.582886 17643 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:53:21.583861 17643 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } }
I20260929 01:53:21.585743 17643 leader_election.cc:290] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 0419fca067cd4c31ad19f304709d4d94 (127.0.71.126:43705), 222b3cc4d7c0489c8760afddb2484746 (127.0.71.125:40265)
I20260929 01:53:21.587266 17440 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0419fca067cd4c31ad19f304709d4d94" is_pre_election: true
I20260929 01:53:21.587831 17440 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c7ddf0bc0cb4650ae96cbe167b2e092 in term 0.
I20260929 01:53:21.589147 17571 leader_election.cc:304] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0419fca067cd4c31ad19f304709d4d94, 8c7ddf0bc0cb4650ae96cbe167b2e092; no voters:
I20260929 01:53:21.590436 17643 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:53:21.590586 17643 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:53:21.590690 17643 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [term 0 FOLLOWER]: Advancing to term 1
W20260929 01:53:21.592921 17420 tablet.cc:2433] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:53:21.594897 17643 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } }
I20260929 01:53:21.597925 17643 leader_election.cc:290] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [CANDIDATE]: Term 1 election: Requested vote from peers 0419fca067cd4c31ad19f304709d4d94 (127.0.71.126:43705), 222b3cc4d7c0489c8760afddb2484746 (127.0.71.125:40265)
I20260929 01:53:21.591210 17552 master_runner.cc:187] Error getting master registration for 127.0.71.125:40265: OK, instance_id { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" instance_seqno: 1790646800905698 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
W20260929 01:53:21.598738 17654 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260929 01:53:21.598755 17441 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0419fca067cd4c31ad19f304709d4d94"
I20260929 01:53:21.598549 17552 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:53:21.598929 17654 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:53:21.599027 17441 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:53:21.602598 17441 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c7ddf0bc0cb4650ae96cbe167b2e092 in term 1.
I20260929 01:53:21.603185 17571 leader_election.cc:304] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0419fca067cd4c31ad19f304709d4d94, 8c7ddf0bc0cb4650ae96cbe167b2e092; no voters:
I20260929 01:53:21.603631 17643 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:53:21.612175 17643 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [term 1 LEADER]: Becoming Leader. State: Replica: 8c7ddf0bc0cb4650ae96cbe167b2e092, State: Running, Role: LEADER
I20260929 01:53:21.612792 17643 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } }
I20260929 01:53:21.616334 17656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c7ddf0bc0cb4650ae96cbe167b2e092. Latest consensus state: current_term: 1 leader_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:21.616693 17656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [sys.catalog]: This master's current role is: LEADER
I20260929 01:53:21.623620 17658 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:53:21.625978 17658 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:53:21.629388 17542 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } attempt: 5
Sep 29 01:53:21 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646800, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:53:21.650574 17441 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [term 1 FOLLOWER]: Refusing update from remote peer 8c7ddf0bc0cb4650ae96cbe167b2e092: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:53:21.651826 17656 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:53:21.659009 17542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746: Bootstrap starting.
I20260929 01:53:21.672585 17542 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746: Neither blocks nor log segments found. Creating new log.
I20260929 01:53:21.675870 17629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c7ddf0bc0cb4650ae96cbe167b2e092. Latest consensus state: current_term: 1 leader_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:21.676177 17629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:21.676092 17542 log.cc:826] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746: Log is configured to *not* fsync() on all Append() calls
I20260929 01:53:21.680012 17659 mvcc.cc:204] Tried to move back new op lower bound from 7334489299545673728 to 7334489299411369984. Current Snapshot: MvccSnapshot[applied={T|T < 7334489299545673728}]
I20260929 01:53:21.684226 17660 mvcc.cc:204] Tried to move back new op lower bound from 7334489299545673728 to 7334489299411369984. Current Snapshot: MvccSnapshot[applied={T|T < 7334489299545673728}]
I20260929 01:53:21.684921 17656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:21.685156 17656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [sys.catalog]: This master's current role is: LEADER
I20260929 01:53:21.687667 17658 catalog_manager.cc:1396] Generated new cluster ID: 1e3f6d40824c4de3a0f91c80f6947064
I20260929 01:53:21.687772 17658 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:53:21.696564 17542 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746: No bootstrap required, opened a new log
I20260929 01:53:21.697091 17629 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:21.697319 17629 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:21.713517 17542 raft_consensus.cc:359] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } }
I20260929 01:53:21.714272 17542 raft_consensus.cc:385] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:53:21.714561 17542 raft_consensus.cc:740] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 222b3cc4d7c0489c8760afddb2484746, State: Initialized, Role: FOLLOWER
I20260929 01:53:21.720356 17542 consensus_queue.cc:260] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } }
I20260929 01:53:21.730723 17658 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 01:53:21.732554 17658 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:53:21.738147 17516 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "222b3cc4d7c0489c8760afddb2484746"
I20260929 01:53:21.738817 17517 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "222b3cc4d7c0489c8760afddb2484746" is_pre_election: true
I20260929 01:53:21.739701 17515 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:53:21.743492 17515 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [term 1 FOLLOWER]: Refusing update from remote peer 8c7ddf0bc0cb4650ae96cbe167b2e092: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:53:21.744417 17656 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [LEADER]: Connected to new peer: Peer: permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:53:21.745499 17517 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8c7ddf0bc0cb4650ae96cbe167b2e092 in term 1.
I20260929 01:53:21.749082 17516 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8c7ddf0bc0cb4650ae96cbe167b2e092 in term 1.
I20260929 01:53:21.756776 17658 catalog_manager.cc:6219] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092: Generated new TSK 0
I20260929 01:53:21.757854 17658 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:53:21.759609 17542 sys_catalog.cc:565] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:53:21.774371 17666 sys_catalog.cc:455] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:21.774712 17666 sys_catalog.cc:458] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:21.775100 17670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:21.775364 17670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:21.780771 17665 sys_catalog.cc:455] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c7ddf0bc0cb4650ae96cbe167b2e092. Latest consensus state: current_term: 1 leader_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:21.781134 17665 sys_catalog.cc:458] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:21.782601 17643 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:21.782871 17643 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [sys.catalog]: This master's current role is: LEADER
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:21.855443 17684 catalog_manager.cc:1308] Loaded cluster ID: 1e3f6d40824c4de3a0f91c80f6947064
I20260929 01:53:21.855628 17684 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746: loading cluster ID for follower catalog manager: success
I20260929 01:53:21.859606 17684 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746: acquiring CA information for follower catalog manager: success
I20260929 01:53:21.862320 17684 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:21.908277 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:33449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:53:22.380638 17688 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:22.380975 17688 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:22.381141 17688 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:22.389788 17688 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:22.390029 17688 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:53:22.402977 17688 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:22.404690 17688 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:22.406113 17688 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:22.415881 17694 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:22.415881 17693 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:22.416955 17696 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:22.417274 17688 server_base.cc:1061] running on GCE node
I20260929 01:53:22.417892 17688 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:22.418861 17688 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:22.420081 17688 hybrid_clock.cc:648] HybridClock initialized: now 1790646802420041 us; error 28 us; skew 500 ppm
I20260929 01:53:22.420378 17688 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:22 dist-test-slave-vz23 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:22.423911 17688 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:53:22.425633 17688 webserver.cc:533] Webserver started at http://127.0.71.65:40201/ using document root <none> and password file <none>
I20260929 01:53:22.426386 17688 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:22.426514 17688 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:22.426893 17688 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:53:22.429451 17688 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "2801d4e8cd0e4abd87fb76207115b8de"
format_stamp: "Formatted at 2026-09-29 01:53:22 on dist-test-slave-vz23"
I20260929 01:53:22.430251 17688 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "2801d4e8cd0e4abd87fb76207115b8de"
format_stamp: "Formatted at 2026-09-29 01:53:22 on dist-test-slave-vz23"
I20260929 01:53:22.435885 17688 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:53:22.439602 17703 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:22.441071 17688 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 01:53:22.441251 17688 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "2801d4e8cd0e4abd87fb76207115b8de"
format_stamp: "Formatted at 2026-09-29 01:53:22 on dist-test-slave-vz23"
I20260929 01:53:22.441438 17688 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:22.476150 17688 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:22.485558 17688 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:22.485877 17688 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:22.486863 17688 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:22.488415 17688 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:22.488538 17688 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 01:53:22.488683 17688 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:22.488757 17688 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:22.523406 17688 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:43381
I20260929 01:53:22.523509 17815 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:43381 every 8 connection(s)
I20260929 01:53:22.525000 17688 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:53:22.534818 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 17688
I20260929 01:53:22.535041 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
Sep 29 01:53:22 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:53:22 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:53:22 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646802, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:53:22.556718 17818 heartbeater.cc:344] Connected to a master server at 127.0.71.125:40265
I20260929 01:53:22.557138 17818 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:22.558254 17818 heartbeater.cc:507] Master 127.0.71.125:40265 requested a full tablet report, sending...
I20260929 01:53:22.559541 17640 catalog_manager.cc:1308] Loaded cluster ID: 1e3f6d40824c4de3a0f91c80f6947064
I20260929 01:53:22.559625 17640 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: loading cluster ID for follower catalog manager: success
I20260929 01:53:22.560863 17507 ts_manager.cc:194] Registered new tserver with Master: 2801d4e8cd0e4abd87fb76207115b8de (127.0.71.65:43381)
I20260929 01:53:22.563819 17817 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43705
I20260929 01:53:22.563882 17640 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: acquiring CA information for follower catalog manager: success
I20260929 01:53:22.563988 17817 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:22.564352 17816 heartbeater.cc:344] Connected to a master server at 127.0.71.124:34639
I20260929 01:53:22.564528 17816 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:22.564571 17817 heartbeater.cc:507] Master 127.0.71.126:43705 requested a full tablet report, sending...
I20260929 01:53:22.565104 17816 heartbeater.cc:507] Master 127.0.71.124:34639 requested a full tablet report, sending...
I20260929 01:53:22.566406 17431 ts_manager.cc:194] Registered new tserver with Master: 2801d4e8cd0e4abd87fb76207115b8de (127.0.71.65:43381)
I20260929 01:53:22.566861 17584 ts_manager.cc:194] Registered new tserver with Master: 2801d4e8cd0e4abd87fb76207115b8de (127.0.71.65:43381)
I20260929 01:53:22.567850 17640 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:53:22.569057 17584 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:53467
WARNING: no policy specified for kudu/127.0.71.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:22.671867 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.66:0
--local_ip_for_outbound_sockets=127.0.71.66
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:33449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:53:23.009860 17830 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:23.010176 17830 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:23.010288 17830 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:23.019217 17830 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:23.019423 17830 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.66
I20260929 01:53:23.030552 17830 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.71.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:23.032258 17830 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:23.033711 17830 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:23.044075 17835 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:23.044039 17836 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:23.045086 17838 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:23.045786 17830 server_base.cc:1061] running on GCE node
I20260929 01:53:23.046420 17830 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:23.047492 17830 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:23.048738 17830 hybrid_clock.cc:648] HybridClock initialized: now 1790646803048695 us; error 35 us; skew 500 ppm
I20260929 01:53:23.049098 17830 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:23 dist-test-slave-vz23 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:23.052330 17830 init.cc:377] Logged in from keytab as kudu/127.0.71.66@KRBTEST.COM (short username kudu)
I20260929 01:53:23.053995 17830 webserver.cc:533] Webserver started at http://127.0.71.66:38163/ using document root <none> and password file <none>
I20260929 01:53:23.054692 17830 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:23.054831 17830 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:23.055181 17830 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:53:23.057669 17830 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/data/instance:
uuid: "d711f64efd3742f9919b75f0c4353415"
format_stamp: "Formatted at 2026-09-29 01:53:23 on dist-test-slave-vz23"
I20260929 01:53:23.058454 17830 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/wal/instance:
uuid: "d711f64efd3742f9919b75f0c4353415"
format_stamp: "Formatted at 2026-09-29 01:53:23 on dist-test-slave-vz23"
I20260929 01:53:23.063907 17830 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 01:53:23.067306 17845 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:23.068779 17830 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:53:23.068975 17830 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/wal
uuid: "d711f64efd3742f9919b75f0c4353415"
format_stamp: "Formatted at 2026-09-29 01:53:23 on dist-test-slave-vz23"
I20260929 01:53:23.069166 17830 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:23.087740 17830 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:23.091478 17830 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:23.091773 17830 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:23.092787 17830 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:23.094435 17830 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:23.094538 17830 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:23.094691 17830 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:23.094765 17830 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:23.130496 17830 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.66:46437
I20260929 01:53:23.130549 17957 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.66:46437 every 8 connection(s)
I20260929 01:53:23.131903 17830 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
I20260929 01:53:23.134006 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 17830
I20260929 01:53:23.134222 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/wal/instance
Sep 29 01:53:23 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:53:23 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:53:23 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:53:23.164361 17959 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43705
I20260929 01:53:23.164384 17958 heartbeater.cc:344] Connected to a master server at 127.0.71.124:34639
I20260929 01:53:23.164608 17960 heartbeater.cc:344] Connected to a master server at 127.0.71.125:40265
I20260929 01:53:23.164921 17959 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:23.164914 17960 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:23.165894 17960 heartbeater.cc:507] Master 127.0.71.125:40265 requested a full tablet report, sending...
I20260929 01:53:23.165921 17959 heartbeater.cc:507] Master 127.0.71.126:43705 requested a full tablet report, sending...
I20260929 01:53:23.164914 17958 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:23.167258 17958 heartbeater.cc:507] Master 127.0.71.124:34639 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.71.67@KRBTEST.COM; defaulting to no policy
I20260929 01:53:23.167701 17507 ts_manager.cc:194] Registered new tserver with Master: d711f64efd3742f9919b75f0c4353415 (127.0.71.66:46437)
I20260929 01:53:23.167836 17431 ts_manager.cc:194] Registered new tserver with Master: d711f64efd3742f9919b75f0c4353415 (127.0.71.66:46437)
I20260929 01:53:23.168184 17584 ts_manager.cc:194] Registered new tserver with Master: d711f64efd3742f9919b75f0c4353415 (127.0.71.66:46437)
I20260929 01:53:23.169196 17584 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.66@KRBTEST.COM'} at 127.0.71.66:46205
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:23.239349 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.67:0
--local_ip_for_outbound_sockets=127.0.71.67
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:33449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:23.572167 17816 heartbeater.cc:499] Master 127.0.71.124:34639 was elected leader, sending a full tablet report...
W20260929 01:53:23.581491 17972 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:23.581789 17972 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:23.581912 17972 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:23.590310 17972 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:23.590513 17972 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.67
I20260929 01:53:23.601361 17972 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.71.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:23.602928 17972 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:23.604241 17972 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:23.614573 17977 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:23.614969 17980 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:23.615097 17978 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:23.615537 17972 server_base.cc:1061] running on GCE node
I20260929 01:53:23.616150 17972 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:23.617084 17972 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:23.618299 17972 hybrid_clock.cc:648] HybridClock initialized: now 1790646803618233 us; error 56 us; skew 500 ppm
I20260929 01:53:23.618659 17972 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:23 dist-test-slave-vz23 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:23.622037 17972 init.cc:377] Logged in from keytab as kudu/127.0.71.67@KRBTEST.COM (short username kudu)
I20260929 01:53:23.623648 17972 webserver.cc:533] Webserver started at http://127.0.71.67:44273/ using document root <none> and password file <none>
I20260929 01:53:23.624393 17972 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:23.624562 17972 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:23.624925 17972 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:53:23.627406 17972 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/data/instance:
uuid: "d76c59eb7518455a933f8048f8a0386a"
format_stamp: "Formatted at 2026-09-29 01:53:23 on dist-test-slave-vz23"
I20260929 01:53:23.628180 17972 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/wal/instance:
uuid: "d76c59eb7518455a933f8048f8a0386a"
format_stamp: "Formatted at 2026-09-29 01:53:23 on dist-test-slave-vz23"
I20260929 01:53:23.633504 17972 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.002s
I20260929 01:53:23.636950 17987 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:23.638314 17972 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 01:53:23.638507 17972 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/wal
uuid: "d76c59eb7518455a933f8048f8a0386a"
format_stamp: "Formatted at 2026-09-29 01:53:23 on dist-test-slave-vz23"
I20260929 01:53:23.638697 17972 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:23.658388 17972 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:23.662711 17972 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:23.663034 17972 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:23.664031 17972 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:23.665560 17972 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:23.665630 17972 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:23.665731 17972 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:23.665776 17972 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:23.698418 17972 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.67:45071
I20260929 01:53:23.698455 18099 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.67:45071 every 8 connection(s)
I20260929 01:53:23.699862 17972 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
I20260929 01:53:23.700289 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 17972
I20260929 01:53:23.700551 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/wal/instance
Sep 29 01:53:23 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:53:23 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:53:23 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646803, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:53:23.740240 18100 heartbeater.cc:344] Connected to a master server at 127.0.71.124:34639
I20260929 01:53:23.740813 18100 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:23.740831 18101 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43705
I20260929 01:53:23.741477 18101 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:23.741715 18102 heartbeater.cc:344] Connected to a master server at 127.0.71.125:40265
I20260929 01:53:23.741868 18102 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:23.742035 18100 heartbeater.cc:507] Master 127.0.71.124:34639 requested a full tablet report, sending...
I20260929 01:53:23.742097 18101 heartbeater.cc:507] Master 127.0.71.126:43705 requested a full tablet report, sending...
I20260929 01:53:23.742401 18102 heartbeater.cc:507] Master 127.0.71.125:40265 requested a full tablet report, sending...
I20260929 01:53:23.743631 17584 ts_manager.cc:194] Registered new tserver with Master: d76c59eb7518455a933f8048f8a0386a (127.0.71.67:45071)
I20260929 01:53:23.743718 17507 ts_manager.cc:194] Registered new tserver with Master: d76c59eb7518455a933f8048f8a0386a (127.0.71.67:45071)
I20260929 01:53:23.744412 17431 ts_manager.cc:194] Registered new tserver with Master: d76c59eb7518455a933f8048f8a0386a (127.0.71.67:45071)
I20260929 01:53:23.744781 17584 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.67@KRBTEST.COM'} at 127.0.71.67:55615
I20260929 01:53:23.763029 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:53:23.770583 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:53:23.770807 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 17399
I20260929 01:53:23.771705 17461 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:53:23.884810 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 17399
I20260929 01:53:23.906881 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:43705
--webserver_interface=127.0.71.126
--webserver_port=40491
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:33449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:53:24.171145 17959 heartbeater.cc:646] Failed to heartbeat to 127.0.71.126:43705 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.71.126:43705: connect: Connection refused (error 111)
I20260929 01:53:24.171514 17958 heartbeater.cc:499] Master 127.0.71.124:34639 was elected leader, sending a full tablet report...
W20260929 01:53:24.253158 18112 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:24.253559 18112 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:24.253724 18112 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:24.263391 18112 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:53:24.263554 18112 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:24.263703 18112 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:53:24.263788 18112 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:53:24.276718 18112 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:43705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=40491
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:24.278728 18112 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:24.280339 18112 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:24.290253 18119 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:24.290226 18118 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:24.291209 18121 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:24.291900 18112 server_base.cc:1061] running on GCE node
I20260929 01:53:24.292817 18112 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:24.294488 18112 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:24.295763 18112 hybrid_clock.cc:648] HybridClock initialized: now 1790646804295703 us; error 79 us; skew 500 ppm
I20260929 01:53:24.296238 18112 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:24 dist-test-slave-vz23 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:24.300328 18112 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:53:24.302232 18112 webserver.cc:533] Webserver started at http://127.0.71.126:40491/ using document root <none> and password file <none>
I20260929 01:53:24.303052 18112 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:24.303215 18112 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:24.309512 18112 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:53:24.312704 18128 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:24.314100 18112 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:53:24.314283 18112 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "0419fca067cd4c31ad19f304709d4d94"
format_stamp: "Formatted at 2026-09-29 01:53:20 on dist-test-slave-vz23"
I20260929 01:53:24.315030 18112 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:24.345609 18112 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:24.349642 18112 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:24.349929 18112 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:24.371835 18112 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:43705
I20260929 01:53:24.371860 18179 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:43705 every 8 connection(s)
I20260929 01:53:24.373605 18112 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:53:24.378789 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 18112
I20260929 01:53:24.379132 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 17688
I20260929 01:53:24.381770 18180 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:53:24.385172 18180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: Bootstrap starting.
Sep 29 01:53:24 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646799, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 01:53:24.398614 17811 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:53:24.411620 18180 log.cc:826] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: Log is configured to *not* fsync() on all Append() calls
I20260929 01:53:24.424341 18180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:53:24.425040 18180 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: Bootstrap complete.
I20260929 01:53:24.433804 18180 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } }
I20260929 01:53:24.434772 18180 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0419fca067cd4c31ad19f304709d4d94, State: Initialized, Role: FOLLOWER
I20260929 01:53:24.435490 18180 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } }
I20260929 01:53:24.436486 18188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:24.436839 18188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:24.437774 18180 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:53:24.438645 18188 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8c7ddf0bc0cb4650ae96cbe167b2e092. Latest consensus state: current_term: 1 leader_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:24.438961 18188 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:24.450641 18199 catalog_manager.cc:1308] Loaded cluster ID: 1e3f6d40824c4de3a0f91c80f6947064
I20260929 01:53:24.450794 18199 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: loading cluster ID for follower catalog manager: success
I20260929 01:53:24.455123 18199 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: acquiring CA information for follower catalog manager: success
Sep 29 01:53:24 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646804, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:53:24.458367 18199 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:53:24.539605 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 17688
I20260929 01:53:24.562942 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:43381
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--webserver_port=40201
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:33449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:24.747372 18100 heartbeater.cc:499] Master 127.0.71.124:34639 was elected leader, sending a full tablet report...
I20260929 01:53:24.758332 18145 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d76c59eb7518455a933f8048f8a0386a" instance_seqno: 1790646803690648) as {username='kudu', principal='kudu/127.0.71.67@KRBTEST.COM'} at 127.0.71.67:36967; Asking this server to re-register.
I20260929 01:53:24.759150 18101 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:24.760843 18145 ts_manager.cc:194] Registered new tserver with Master: d76c59eb7518455a933f8048f8a0386a (127.0.71.67:45071)
W20260929 01:53:24.983261 18204 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:24.983603 18204 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:24.983711 18204 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:24.993393 18204 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:24.993599 18204 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:53:25.008978 18204 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:43381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=40201
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:25.011193 18204 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:25.013079 18204 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:25.024042 18210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:25.024138 18211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:25.025095 18213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:25.025890 18204 server_base.cc:1061] running on GCE node
I20260929 01:53:25.026556 18204 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:25.027578 18204 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:25.028815 18204 hybrid_clock.cc:648] HybridClock initialized: now 1790646805028766 us; error 65 us; skew 500 ppm
I20260929 01:53:25.029167 18204 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:25 dist-test-slave-vz23 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:25.032871 18204 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:53:25.034914 18204 webserver.cc:533] Webserver started at http://127.0.71.65:40201/ using document root <none> and password file <none>
I20260929 01:53:25.036022 18204 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:25.036191 18204 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:25.044929 18204 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 01:53:25.048841 18220 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:25.051270 18204 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260929 01:53:25.051488 18204 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "2801d4e8cd0e4abd87fb76207115b8de"
format_stamp: "Formatted at 2026-09-29 01:53:22 on dist-test-slave-vz23"
I20260929 01:53:25.052681 18204 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:25.087733 18204 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:25.093348 18204 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:25.093676 18204 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:25.094767 18204 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:25.096585 18204 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:25.096678 18204 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:25.096805 18204 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:25.097136 18204 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:25.133612 18204 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:43381
I20260929 01:53:25.133651 18332 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:43381 every 8 connection(s)
I20260929 01:53:25.135385 18204 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:53:25.140544 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 18204
Sep 29 01:53:25 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:53:25 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:53:25 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:53:25.166086 18333 heartbeater.cc:344] Connected to a master server at 127.0.71.124:34639
I20260929 01:53:25.166028 18334 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43705
I20260929 01:53:25.166501 18333 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:25.166501 18334 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:25.167697 18333 heartbeater.cc:507] Master 127.0.71.124:34639 requested a full tablet report, sending...
I20260929 01:53:25.167719 18334 heartbeater.cc:507] Master 127.0.71.126:43705 requested a full tablet report, sending...
I20260929 01:53:25.169883 17584 ts_manager.cc:194] Re-registered known tserver with Master: 2801d4e8cd0e4abd87fb76207115b8de (127.0.71.65:43381)
I20260929 01:53:25.171054 17584 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:60897
I20260929 01:53:25.175102 18145 ts_manager.cc:194] Registered new tserver with Master: 2801d4e8cd0e4abd87fb76207115b8de (127.0.71.65:43381)
I20260929 01:53:25.177809 18335 heartbeater.cc:344] Connected to a master server at 127.0.71.125:40265
I20260929 01:53:25.177983 18335 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:25.178337 18335 heartbeater.cc:507] Master 127.0.71.125:40265 requested a full tablet report, sending...
I20260929 01:53:25.179675 17507 ts_manager.cc:194] Re-registered known tserver with Master: 2801d4e8cd0e4abd87fb76207115b8de (127.0.71.65:43381)
I20260929 01:53:25.186704 17959 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43705
I20260929 01:53:25.187879 18145 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d711f64efd3742f9919b75f0c4353415" instance_seqno: 1790646803122209) as {username='kudu', principal='kudu/127.0.71.66@KRBTEST.COM'} at 127.0.71.66:51481; Asking this server to re-register.
I20260929 01:53:25.188485 17959 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:25.189581 18145 ts_manager.cc:194] Registered new tserver with Master: d711f64efd3742f9919b75f0c4353415 (127.0.71.66:46437)
I20260929 01:53:25.189906 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 18204
W20260929 01:53:25.191059 18346 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:25.187099 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:42876 (local address 127.0.71.65:43381)
0929 01:53:25.188065 (+ 966us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:25.188075 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:25.188105 (+ 30us) server_negotiation.cc:408] Connection header received
0929 01:53:25.188173 (+ 68us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:53:25.188179 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:53:25.188526 (+ 347us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:53:25.188748 (+ 222us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:53:25.189707 (+ 959us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:42876: BlockingRecv error: recv got EOF from 127.0.0.1:42876 (error 108)
Metrics: {"server-negotiator.queue_time_us":816,"thread_start_us":1019,"threads_started":1}
W20260929 01:53:25.192355 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 18204: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:43381: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:53:25.192528 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 18204
I20260929 01:53:25.209416 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:43381
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--webserver_port=40201
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:33449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:53:25.552325 18348 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:25.552738 18348 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:25.552899 18348 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:53:25.552994 18348 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:25.561697 18348 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:25.561954 18348 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:53:25.573019 18348 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:43381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=40201
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:25.574880 18348 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:25.576342 18348 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:25.586673 18354 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:25.586643 18353 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:25.588429 18356 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:25.589269 18348 server_base.cc:1061] running on GCE node
I20260929 01:53:25.589959 18348 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:25.590977 18348 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:25.592224 18348 hybrid_clock.cc:648] HybridClock initialized: now 1790646805592163 us; error 38 us; skew 500 ppm
I20260929 01:53:25.592623 18348 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:25 dist-test-slave-vz23 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646805, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:25.596189 18348 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:53:25.598119 18348 webserver.cc:533] Webserver started at http://127.0.71.65:40201/ using document root <none> and password file <none>
I20260929 01:53:25.598901 18348 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:25.599028 18348 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:25.605002 18348 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:53:25.607995 18363 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:25.609634 18348 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:53:25.609808 18348 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "2801d4e8cd0e4abd87fb76207115b8de"
format_stamp: "Formatted at 2026-09-29 01:53:22 on dist-test-slave-vz23"
I20260929 01:53:25.610577 18348 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:25.631090 18348 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:25.635183 18348 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:25.635481 18348 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:25.636554 18348 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:25.638269 18348 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:25.638377 18348 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:25.638538 18348 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:25.638615 18348 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:53:25.639077 18348 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:53:25.639168 18348 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:53:25.639249 18348 fault_injection.cc:59] injecting fault for kudu (pid 18348): FLAGS_fault_before_start (process will exit)
I20260929 01:53:25.671844 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:43381
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--webserver_port=40201
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:33449
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:53:26.007241 18371 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:26.007567 18371 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:26.008030 18371 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:26.016876 18371 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:26.017184 18371 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:53:26.028052 18371 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33449
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:43381
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=40201
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:26.029726 18371 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:26.031031 18371 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:26.040937 18377 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:26.040998 18378 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:26.041031 18380 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:26.042148 18371 server_base.cc:1061] running on GCE node
I20260929 01:53:26.042732 18371 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:26.043712 18371 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:26.044975 18371 hybrid_clock.cc:648] HybridClock initialized: now 1790646806044910 us; error 56 us; skew 500 ppm
I20260929 01:53:26.045307 18371 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:26 dist-test-slave-vz23 krb5kdc[17383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:26.048595 18371 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:53:26.050186 18371 webserver.cc:533] Webserver started at http://127.0.71.65:40201/ using document root <none> and password file <none>
I20260929 01:53:26.050832 18371 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:26.050952 18371 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:26.056249 18371 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:53:26.058957 18387 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:26.060233 18371 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:53:26.060411 18371 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "2801d4e8cd0e4abd87fb76207115b8de"
format_stamp: "Formatted at 2026-09-29 01:53:22 on dist-test-slave-vz23"
I20260929 01:53:26.061231 18371 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:26.101809 18371 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:26.106139 18371 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:26.106380 18371 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:26.107342 18371 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:26.108965 18371 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:26.109055 18371 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:26.109136 18371 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:26.109190 18371 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:26.143841 18371 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:43381
I20260929 01:53:26.143978 18499 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:43381 every 8 connection(s)
I20260929 01:53:26.145411 18371 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:53:26.153910 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 18371
I20260929 01:53:26.154264 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 18371
Sep 29 01:53:26 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:53:26 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
W20260929 01:53:26.168099 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 18371: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:43381: server requires authentication, but client does not have Kerberos credentials available
Sep 29 01:53:26 dist-test-slave-vz23 krb5kdc[17383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646806, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:53:26.168257 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 18371
W20260929 01:53:26.186367 18509 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:26.151322 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:53819 (local address 127.0.71.125:40265)
0929 01:53:26.153037 (+ 1715us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:26.153041 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:26.153056 (+ 15us) server_negotiation.cc:408] Connection header received
0929 01:53:26.153103 (+ 47us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:53:26.153108 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:53:26.153162 (+ 54us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:53:26.153297 (+ 135us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:53:26.160598 (+ 7301us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:53:26.163847 (+ 3249us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:53:26.165922 (+ 2075us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:53:26.166479 (+ 557us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0929 01:53:26.185553 (+ 19074us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:53819: BlockingRecv error: recv got EOF from 127.0.71.65:53819 (error 108)
Metrics: {"server-negotiator.queue_time_us":1635,"thread_start_us":1499,"threads_started":1}
W20260929 01:53:26.186615 18504 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:26.148714 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:45979 (local address 127.0.71.126:43705)
0929 01:53:26.149226 (+ 512us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:26.149234 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:26.150519 (+ 1285us) server_negotiation.cc:408] Connection header received
0929 01:53:26.152755 (+ 2236us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:53:26.152788 (+ 33us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:53:26.152909 (+ 121us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:53:26.153019 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:53:26.156988 (+ 3969us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:53:26.158831 (+ 1843us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:53:26.161169 (+ 2338us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:53:26.162218 (+ 1049us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0929 01:53:26.165020 (+ 2802us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0929 01:53:26.165069 (+ 49us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0929 01:53:26.165075 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0929 01:53:26.165111 (+ 36us) server_negotiation.cc:923] calling sasl_server_start()
0929 01:53:26.169260 (+ 4149us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0929 01:53:26.185702 (+ 16442us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:45979: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":382,"thread_start_us":285,"threads_started":1}
W20260929 01:53:26.186615 18505 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:26.148716 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:54661 (local address 127.0.71.124:34639)
0929 01:53:26.149386 (+ 670us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:26.149390 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:26.150981 (+ 1591us) server_negotiation.cc:408] Connection header received
0929 01:53:26.153371 (+ 2390us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:53:26.153377 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:53:26.153461 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:53:26.153561 (+ 100us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:53:26.156910 (+ 3349us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:53:26.157879 (+ 969us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:53:26.159066 (+ 1187us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:53:26.159570 (+ 504us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0929 01:53:26.164559 (+ 4989us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0929 01:53:26.164605 (+ 46us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0929 01:53:26.164612 (+ 7us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0929 01:53:26.164653 (+ 41us) server_negotiation.cc:923] calling sasl_server_start()
0929 01:53:26.168396 (+ 3743us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0929 01:53:26.185696 (+ 17300us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:54661: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":447,"threads_started":1}
I20260929 01:53:26.186990 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 17830
W20260929 01:53:26.189682 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 17830: Not authorized: Client connection negotiation failed: client connection to 127.0.71.66:46437: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:53:26.189826 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 17830
I20260929 01:53:26.205775 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 17972
W20260929 01:53:26.208361 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 17972: Not authorized: Client connection negotiation failed: client connection to 127.0.71.67:45071: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:53:26.208546 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 17972
I20260929 01:53:26.224089 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 18112
W20260929 01:53:26.226167 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 18112: Not authorized: Client connection negotiation failed: client connection to 127.0.71.126:43705: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:53:26.226275 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 18112
I20260929 01:53:26.246417 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 17475
I20260929 01:53:26.247577 17537 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:53:26.357339 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 17475
I20260929 01:53:26.378513 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 17552
I20260929 01:53:26.379849 17614 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:53:26.493431 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 17552
2026-09-29T01:53:26Z chronyd exiting
Sep 29 01:53:26 dist-test-slave-vz23 krb5kdc[18517](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:53:26 dist-test-slave-vz23 krb5kdc[18517](info): set up 2 sockets
Sep 29 01:53:26 dist-test-slave-vz23 krb5kdc[18517](info): commencing operation
krb5kdc: starting...
W20260929 01:53:28.537405 285 mini_kdc.cc:121] Time spent starting KDC: real 2.009s user 0.000s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-admin@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-user@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "joe-interloper@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 01:53:28 dist-test-slave-vz23 krb5kdc[18517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:53:28Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:53:28Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.126@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.126@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:28.760149 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:43705
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:44631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:53:29.101336 18533 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:29.101650 18533 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:29.101833 18533 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:29.110596 18533 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:53:29.110718 18533 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:29.110877 18533 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:53:29.110947 18533 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:53:29.122244 18533 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:44631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:43705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:29.123872 18533 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:29.125312 18533 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:29.134039 18539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:29.134181 18538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:29.136687 18533 server_base.cc:1061] running on GCE node
W20260929 01:53:29.137684 18541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:29.138690 18533 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:29.140184 18533 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:29.141443 18533 hybrid_clock.cc:648] HybridClock initialized: now 1790646809141471 us; error 136 us; skew 500 ppm
I20260929 01:53:29.141808 18533 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:29 dist-test-slave-vz23 krb5kdc[18517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:29.145386 18533 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:53:29.147161 18533 webserver.cc:533] Webserver started at http://127.0.71.126:34189/ using document root <none> and password file <none>
I20260929 01:53:29.147897 18533 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:29.148038 18533 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:29.153630 18533 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:53:29.156564 18548 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:29.157949 18533 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:53:29.158128 18533 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "0419fca067cd4c31ad19f304709d4d94"
format_stamp: "Formatted at 2026-09-29 01:53:20 on dist-test-slave-vz23"
I20260929 01:53:29.158859 18533 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:29.199327 18533 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:29.209762 18533 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:29.210067 18533 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:29.231055 18533 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:43705
I20260929 01:53:29.231073 18599 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:43705 every 8 connection(s)
I20260929 01:53:29.232789 18533 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:53:29.239800 18600 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:53:29.243155 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 18533
I20260929 01:53:29.243348 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:53:29.243536 18600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: Bootstrap starting.
I20260929 01:53:29.262426 18600 log.cc:826] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: Log is configured to *not* fsync() on all Append() calls
WARNING: no policy specified for kudu/127.0.71.125@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.125@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:53:29.273018 18600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:53:29.273537 18600 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: Bootstrap complete.
I20260929 01:53:29.280658 18600 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } }
I20260929 01:53:29.281412 18600 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0419fca067cd4c31ad19f304709d4d94, State: Initialized, Role: FOLLOWER
I20260929 01:53:29.282157 18600 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } }
I20260929 01:53:29.283073 18605 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:29.283372 18605 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:29.284263 18600 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:53:29.294971 18616 catalog_manager.cc:1308] Loaded cluster ID: 1e3f6d40824c4de3a0f91c80f6947064
I20260929 01:53:29.295081 18616 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: loading cluster ID for follower catalog manager: success
I20260929 01:53:29.299557 18616 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: acquiring CA information for follower catalog manager: success
I20260929 01:53:29.302387 18616 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Sep 29 01:53:29 dist-test-slave-vz23 krb5kdc[18517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.125 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.125 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:29.328264 18533 master_runner.cc:187] Error getting master registration for 127.0.71.125:40265: Network error: Client connection negotiation failed: client connection to 127.0.71.125:40265: connect: Connection refused (error 111),
I20260929 01:53:29.335326 18533 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
WARNING: no policy specified for HTTP/127.0.71.125@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.125@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.125 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.125 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:29.365128 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:40265
--webserver_interface=127.0.71.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:44631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:53:29.701828 18622 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:29.702206 18622 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:29.702369 18622 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:29.710805 18622 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:53:29.710961 18622 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:29.711097 18622 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:53:29.711180 18622 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:53:29.722420 18622 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:44631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:40265
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:29.724120 18622 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:29.725600 18622 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:29.733860 18627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:29.733834 18628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:29.735596 18630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:29.735574 18622 server_base.cc:1061] running on GCE node
I20260929 01:53:29.736533 18622 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:29.737931 18622 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:29.739143 18622 hybrid_clock.cc:648] HybridClock initialized: now 1790646809739110 us; error 32 us; skew 500 ppm
I20260929 01:53:29.739465 18622 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:29 dist-test-slave-vz23 krb5kdc[18517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646809, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:29.742930 18622 init.cc:377] Logged in from keytab as kudu/127.0.71.125@KRBTEST.COM (short username kudu)
I20260929 01:53:29.744674 18622 webserver.cc:533] Webserver started at http://127.0.71.125:44709/ using document root <none> and password file <none>
I20260929 01:53:29.745383 18622 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:29.745486 18622 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:29.750937 18622 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:53:29.753799 18637 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:29.755172 18622 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:53:29.755376 18622 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "222b3cc4d7c0489c8760afddb2484746"
format_stamp: "Formatted at 2026-09-29 01:53:20 on dist-test-slave-vz23"
I20260929 01:53:29.756088 18622 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:29.785597 18622 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:29.789898 18622 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:29.790169 18622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:29.810508 18622 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:40265
I20260929 01:53:29.810541 18688 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:40265 every 8 connection(s)
I20260929 01:53:29.812109 18622 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:53:29.818363 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 18622
I20260929 01:53:29.818578 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:53:29.819599 18689 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:53:29.824865 18689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746: Bootstrap starting.
WARNING: no policy specified for kudu/127.0.71.124@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.124@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:53:29.843287 18689 log.cc:826] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746: Log is configured to *not* fsync() on all Append() calls
I20260929 01:53:29.852216 18689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:53:29.852837 18689 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746: Bootstrap complete.
I20260929 01:53:29.859634 18689 raft_consensus.cc:359] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } }
I20260929 01:53:29.860369 18689 raft_consensus.cc:740] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 222b3cc4d7c0489c8760afddb2484746, State: Initialized, Role: FOLLOWER
I20260929 01:53:29.861307 18689 consensus_queue.cc:260] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } }
I20260929 01:53:29.862149 18694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:29.862387 18694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:29.863273 18689 sys_catalog.cc:565] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [sys.catalog]: configured and running, proceeding with master startup.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.124 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.124 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:29.874168 18705 catalog_manager.cc:1308] Loaded cluster ID: 1e3f6d40824c4de3a0f91c80f6947064
I20260929 01:53:29.874356 18705 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746: loading cluster ID for follower catalog manager: success
I20260929 01:53:29.878762 18705 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746: acquiring CA information for follower catalog manager: success
I20260929 01:53:29.881654 18705 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for HTTP/127.0.71.124@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.124@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:53:29.905478 18622 master_runner.cc:187] Error getting master registration for 127.0.71.124:34639: Network error: Client connection negotiation failed: client connection to 127.0.71.124:34639: connect: Connection refused (error 111),
I20260929 01:53:29.910163 18622 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.124 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.124 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:29.929973 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:34639
--webserver_interface=127.0.71.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:44631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:53:30.273222 18712 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:30.273602 18712 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:30.273753 18712 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:30.282335 18712 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:53:30.282487 18712 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:30.282619 18712 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:53:30.282702 18712 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:53:30.295430 18712 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:44631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:34639
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:30.297200 18712 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:30.298621 18712 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:30.306694 18718 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:30.306762 18717 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:30.307910 18720 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:30.308640 18712 server_base.cc:1061] running on GCE node
I20260929 01:53:30.309409 18712 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:30.310788 18712 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:30.312036 18712 hybrid_clock.cc:648] HybridClock initialized: now 1790646810311988 us; error 54 us; skew 500 ppm
I20260929 01:53:30.312387 18712 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:30 dist-test-slave-vz23 krb5kdc[18517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:30.315881 18712 init.cc:377] Logged in from keytab as kudu/127.0.71.124@KRBTEST.COM (short username kudu)
I20260929 01:53:30.317520 18712 webserver.cc:533] Webserver started at http://127.0.71.124:45781/ using document root <none> and password file <none>
I20260929 01:53:30.318229 18712 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:30.318377 18712 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:30.323734 18712 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 01:53:30.326637 18727 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:30.327951 18712 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:53:30.328122 18712 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092"
format_stamp: "Formatted at 2026-09-29 01:53:21 on dist-test-slave-vz23"
I20260929 01:53:30.328992 18712 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:30.348140 18712 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:30.351871 18712 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:30.352151 18712 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:30.373251 18712 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:34639
I20260929 01:53:30.373286 18778 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:34639 every 8 connection(s)
I20260929 01:53:30.374884 18712 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:53:30.381894 18779 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:53:30.383083 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 18712
I20260929 01:53:30.383284 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:53:30.385737 18779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092: Bootstrap starting.
Sep 29 01:53:30 dist-test-slave-vz23 krb5kdc[18517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:53:30.405030 18779 log.cc:826] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092: Log is configured to *not* fsync() on all Append() calls
Sep 29 01:53:30 dist-test-slave-vz23 krb5kdc[18517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:53:30.414270 18779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:53:30.414829 18779 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092: Bootstrap complete.
I20260929 01:53:30.422010 18779 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } }
I20260929 01:53:30.422715 18779 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8c7ddf0bc0cb4650ae96cbe167b2e092, State: Initialized, Role: FOLLOWER
I20260929 01:53:30.423542 18779 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } }
I20260929 01:53:30.424916 18784 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:30.425400 18784 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:30.426021 18779 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 01:53:30 dist-test-slave-vz23 krb5kdc[18517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:53:30 dist-test-slave-vz23 krb5kdc[18517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646810, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:53:30.446931 18795 catalog_manager.cc:1308] Loaded cluster ID: 1e3f6d40824c4de3a0f91c80f6947064
I20260929 01:53:30.447037 18795 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092: loading cluster ID for follower catalog manager: success
I20260929 01:53:30.450855 18795 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092: acquiring CA information for follower catalog manager: success
I20260929 01:53:30.453532 18795 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:53:30.472000 18712 master_runner.cc:264] No leader master found from master 8c7ddf0bc0cb4650ae96cbe167b2e092
I20260929 01:53:30.472151 18712 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:53:31.040203 18798 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:53:31.040503 18798 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } }
I20260929 01:53:31.042197 18798 leader_election.cc:290] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 222b3cc4d7c0489c8760afddb2484746 (127.0.71.125:40265), 8c7ddf0bc0cb4650ae96cbe167b2e092 (127.0.71.124:34639)
I20260929 01:53:31.063578 18664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0419fca067cd4c31ad19f304709d4d94" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "222b3cc4d7c0489c8760afddb2484746" is_pre_election: true
I20260929 01:53:31.064204 18664 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0419fca067cd4c31ad19f304709d4d94 in term 1.
I20260929 01:53:31.064749 18754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0419fca067cd4c31ad19f304709d4d94" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" is_pre_election: true
I20260929 01:53:31.064858 18552 leader_election.cc:304] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0419fca067cd4c31ad19f304709d4d94, 222b3cc4d7c0489c8760afddb2484746; no voters:
I20260929 01:53:31.065256 18754 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0419fca067cd4c31ad19f304709d4d94 in term 1.
I20260929 01:53:31.065292 18798 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 01:53:31.065438 18798 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:53:31.065548 18798 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:53:31.069483 18798 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } }
I20260929 01:53:31.070255 18798 leader_election.cc:290] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [CANDIDATE]: Term 2 election: Requested vote from peers 222b3cc4d7c0489c8760afddb2484746 (127.0.71.125:40265), 8c7ddf0bc0cb4650ae96cbe167b2e092 (127.0.71.124:34639)
I20260929 01:53:31.070725 18754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0419fca067cd4c31ad19f304709d4d94" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092"
I20260929 01:53:31.070729 18664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0419fca067cd4c31ad19f304709d4d94" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "222b3cc4d7c0489c8760afddb2484746"
I20260929 01:53:31.070931 18754 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:53:31.070943 18664 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:53:31.074481 18664 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0419fca067cd4c31ad19f304709d4d94 in term 2.
I20260929 01:53:31.074607 18754 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0419fca067cd4c31ad19f304709d4d94 in term 2.
I20260929 01:53:31.074929 18552 leader_election.cc:304] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0419fca067cd4c31ad19f304709d4d94, 222b3cc4d7c0489c8760afddb2484746; no voters:
I20260929 01:53:31.075261 18798 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:53:31.075654 18798 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [term 2 LEADER]: Becoming Leader. State: Replica: 0419fca067cd4c31ad19f304709d4d94, State: Running, Role: LEADER
I20260929 01:53:31.076118 18798 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } }
I20260929 01:53:31.078930 18803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0419fca067cd4c31ad19f304709d4d94. Latest consensus state: current_term: 2 leader_uuid: "0419fca067cd4c31ad19f304709d4d94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:31.079228 18803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [sys.catalog]: This master's current role is: LEADER
I20260929 01:53:31.079944 18805 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:53:31.082021 18805 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:53:31.082846 18805 catalog_manager.cc:1308] Loaded cluster ID: 1e3f6d40824c4de3a0f91c80f6947064
I20260929 01:53:31.082949 18805 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:53:31.084079 18805 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:53:31.084956 18805 catalog_manager.cc:6230] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: Loaded TSK: 0
I20260929 01:53:31.085959 18805 catalog_manager.cc:1563] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.65@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.65@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:31.182248 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:44631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:31.514098 18664 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [term 2 FOLLOWER]: Refusing update from remote peer 0419fca067cd4c31ad19f304709d4d94: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
W20260929 01:53:31.514446 18810 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:31.514760 18810 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:31.514923 18810 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260929 01:53:31.515200 18803 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:53:31.519172 18814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0419fca067cd4c31ad19f304709d4d94. Latest consensus state: current_term: 2 leader_uuid: "0419fca067cd4c31ad19f304709d4d94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:31.519470 18814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:31.520617 18798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "0419fca067cd4c31ad19f304709d4d94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:31.520895 18798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [sys.catalog]: This master's current role is: LEADER
I20260929 01:53:31.521792 18754 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [term 2 FOLLOWER]: Refusing update from remote peer 0419fca067cd4c31ad19f304709d4d94: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 01:53:31.521795 18814 sys_catalog.cc:455] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "0419fca067cd4c31ad19f304709d4d94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:31.521968 18814 sys_catalog.cc:458] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:31.522622 18803 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260929 01:53:31.525303 18810 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:31.525584 18810 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:53:31.526291 18821 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "0419fca067cd4c31ad19f304709d4d94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:31.526583 18821 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:31.526875 18818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0419fca067cd4c31ad19f304709d4d94. Latest consensus state: current_term: 2 leader_uuid: "0419fca067cd4c31ad19f304709d4d94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:31.527115 18818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:31.537011 18810 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:44631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:31.538523 18810 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:31.539712 18810 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:31.548946 18824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:31.549109 18823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:31.549779 18826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:31.550493 18810 server_base.cc:1061] running on GCE node
I20260929 01:53:31.550968 18810 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:31.551883 18810 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:31.553071 18810 hybrid_clock.cc:648] HybridClock initialized: now 1790646811553036 us; error 24 us; skew 500 ppm
I20260929 01:53:31.553406 18810 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:31 dist-test-slave-vz23 krb5kdc[18517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:31.556777 18810 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:53:31.558430 18810 webserver.cc:533] Webserver started at http://127.0.71.65:41407/ using document root <none> and password file <none>
I20260929 01:53:31.559116 18810 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:31.559258 18810 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:31.564920 18810 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:53:31.567672 18833 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:31.569084 18810 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:53:31.569257 18810 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "2801d4e8cd0e4abd87fb76207115b8de"
format_stamp: "Formatted at 2026-09-29 01:53:22 on dist-test-slave-vz23"
I20260929 01:53:31.569950 18810 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:31.605885 18810 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:31.610072 18810 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:31.610354 18810 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:31.611454 18810 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:31.613121 18810 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:31.613243 18810 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:31.613373 18810 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:31.613471 18810 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:31.648629 18810 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:34225
I20260929 01:53:31.648675 18945 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:34225 every 8 connection(s)
I20260929 01:53:31.650127 18810 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:53:31.653846 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 18810
I20260929 01:53:31.654063 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
Sep 29 01:53:31 dist-test-slave-vz23 krb5kdc[18517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:53:31 dist-test-slave-vz23 krb5kdc[18517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:53:31.681644 18947 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43705
I20260929 01:53:31.682044 18947 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:31.683096 18947 heartbeater.cc:507] Master 127.0.71.126:43705 requested a full tablet report, sending...
I20260929 01:53:31.684036 18946 heartbeater.cc:344] Connected to a master server at 127.0.71.124:34639
I20260929 01:53:31.684181 18946 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:31.684751 18946 heartbeater.cc:507] Master 127.0.71.124:34639 requested a full tablet report, sending...
I20260929 01:53:31.685796 18565 ts_manager.cc:194] Registered new tserver with Master: 2801d4e8cd0e4abd87fb76207115b8de (127.0.71.65:34225)
I20260929 01:53:31.686559 18744 ts_manager.cc:194] Registered new tserver with Master: 2801d4e8cd0e4abd87fb76207115b8de (127.0.71.65:34225)
I20260929 01:53:31.688104 18565 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:44779
WARNING: no policy specified for kudu/127.0.71.66@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.66@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Sep 29 01:53:31 dist-test-slave-vz23 krb5kdc[18517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646811, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:53:31.700346 18948 heartbeater.cc:344] Connected to a master server at 127.0.71.125:40265
I20260929 01:53:31.700573 18948 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:31.700935 18948 heartbeater.cc:507] Master 127.0.71.125:40265 requested a full tablet report, sending...
I20260929 01:53:31.702615 18654 ts_manager.cc:194] Registered new tserver with Master: 2801d4e8cd0e4abd87fb76207115b8de (127.0.71.65:34225)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.66 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.66 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.66@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.66@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.66 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.66 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:31.764155 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.66:0
--local_ip_for_outbound_sockets=127.0.71.66
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:44631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:32.007820 18803 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "0419fca067cd4c31ad19f304709d4d94" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:32.008144 18803 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [sys.catalog]: This master's current role is: LEADER
W20260929 01:53:32.104072 18960 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:32.104388 18960 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:32.104655 18960 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:32.113612 18960 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:32.113889 18960 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.66
I20260929 01:53:32.124861 18960 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:44631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.71.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:32.126515 18960 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:32.127928 18960 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:32.138965 18966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:32.139067 18967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:32.139274 18969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:32.140771 18960 server_base.cc:1061] running on GCE node
I20260929 01:53:32.141419 18960 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:32.142351 18960 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:32.143567 18960 hybrid_clock.cc:648] HybridClock initialized: now 1790646812143528 us; error 32 us; skew 500 ppm
I20260929 01:53:32.143929 18960 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:32 dist-test-slave-vz23 krb5kdc[18517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:32.147681 18960 init.cc:377] Logged in from keytab as kudu/127.0.71.66@KRBTEST.COM (short username kudu)
I20260929 01:53:32.149550 18960 webserver.cc:533] Webserver started at http://127.0.71.66:39181/ using document root <none> and password file <none>
I20260929 01:53:32.150264 18960 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:32.150430 18960 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:32.155933 18960 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:53:32.158744 18976 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:32.160358 18960 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:53:32.160557 18960 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/wal
uuid: "d711f64efd3742f9919b75f0c4353415"
format_stamp: "Formatted at 2026-09-29 01:53:23 on dist-test-slave-vz23"
I20260929 01:53:32.161319 18960 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:32.174957 18960 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:32.178738 18960 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:32.179011 18960 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:32.179991 18960 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:32.181748 18960 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:32.181836 18960 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:32.181980 18960 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:32.182070 18960 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:32.217224 18960 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.66:33363
I20260929 01:53:32.217293 19088 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.66:33363 every 8 connection(s)
I20260929 01:53:32.218851 18960 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
I20260929 01:53:32.226346 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 18960
I20260929 01:53:32.226589 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-1/wal/instance
Sep 29 01:53:32 dist-test-slave-vz23 krb5kdc[18517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:53:32 dist-test-slave-vz23 krb5kdc[18517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:53:32 dist-test-slave-vz23 krb5kdc[18517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:53:32.262447 19091 heartbeater.cc:344] Connected to a master server at 127.0.71.125:40265
I20260929 01:53:32.262919 19091 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:32.264107 19091 heartbeater.cc:507] Master 127.0.71.125:40265 requested a full tablet report, sending...
I20260929 01:53:32.266507 18654 ts_manager.cc:194] Registered new tserver with Master: d711f64efd3742f9919b75f0c4353415 (127.0.71.66:33363)
I20260929 01:53:32.268798 19089 heartbeater.cc:344] Connected to a master server at 127.0.71.124:34639
I20260929 01:53:32.268970 19090 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43705
I20260929 01:53:32.268977 19089 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:32.269122 19090 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:32.269706 19090 heartbeater.cc:507] Master 127.0.71.126:43705 requested a full tablet report, sending...
I20260929 01:53:32.269727 19089 heartbeater.cc:507] Master 127.0.71.124:34639 requested a full tablet report, sending...
I20260929 01:53:32.271029 18744 ts_manager.cc:194] Registered new tserver with Master: d711f64efd3742f9919b75f0c4353415 (127.0.71.66:33363)
WARNING: no policy specified for kudu/127.0.71.67@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.67@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:53:32.271574 18565 ts_manager.cc:194] Registered new tserver with Master: d711f64efd3742f9919b75f0c4353415 (127.0.71.66:33363)
I20260929 01:53:32.272641 18565 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.66@KRBTEST.COM'} at 127.0.71.66:52619
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.67 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.67 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.67@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.67@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.67 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.67 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:32.343297 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.67:0
--local_ip_for_outbound_sockets=127.0.71.67
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:44631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:32.691282 18947 heartbeater.cc:499] Master 127.0.71.126:43705 was elected leader, sending a full tablet report...
W20260929 01:53:32.694633 19103 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:32.694947 19103 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:32.695127 19103 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:32.703914 19103 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:32.704159 19103 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.67
I20260929 01:53:32.715296 19103 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:44631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.71.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:32.716940 19103 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:32.718398 19103 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:32.727980 19109 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:32.728276 19108 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:32.728971 19111 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:32.730319 19103 server_base.cc:1061] running on GCE node
I20260929 01:53:32.731005 19103 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:32.731997 19103 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:32.733227 19103 hybrid_clock.cc:648] HybridClock initialized: now 1790646812733168 us; error 77 us; skew 500 ppm
I20260929 01:53:32.733572 19103 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:32 dist-test-slave-vz23 krb5kdc[18517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:32.737596 19103 init.cc:377] Logged in from keytab as kudu/127.0.71.67@KRBTEST.COM (short username kudu)
I20260929 01:53:32.739554 19103 webserver.cc:533] Webserver started at http://127.0.71.67:34679/ using document root <none> and password file <none>
I20260929 01:53:32.740237 19103 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:32.740345 19103 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:32.745909 19103 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:53:32.748776 19118 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:32.750272 19103 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:53:32.750433 19103 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/wal
uuid: "d76c59eb7518455a933f8048f8a0386a"
format_stamp: "Formatted at 2026-09-29 01:53:23 on dist-test-slave-vz23"
I20260929 01:53:32.751163 19103 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:32.768276 19103 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:32.772351 19103 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:32.772698 19103 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:32.773640 19103 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:32.775410 19103 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:32.775527 19103 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:32.775647 19103 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:32.775771 19103 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:32.811339 19103 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.67:44367
I20260929 01:53:32.811384 19230 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.67:44367 every 8 connection(s)
I20260929 01:53:32.812934 19103 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
I20260929 01:53:32.815917 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 19103
I20260929 01:53:32.816145 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-2/wal/instance
Sep 29 01:53:32 dist-test-slave-vz23 krb5kdc[18517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:53:32 dist-test-slave-vz23 krb5kdc[18517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:53:32 dist-test-slave-vz23 krb5kdc[18517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646812, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:53:32.863411 19231 heartbeater.cc:344] Connected to a master server at 127.0.71.124:34639
I20260929 01:53:32.863819 19231 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:32.864881 19231 heartbeater.cc:507] Master 127.0.71.124:34639 requested a full tablet report, sending...
I20260929 01:53:32.865852 19232 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43705
I20260929 01:53:32.866005 19232 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:32.866575 19232 heartbeater.cc:507] Master 127.0.71.126:43705 requested a full tablet report, sending...
I20260929 01:53:32.866601 19233 heartbeater.cc:344] Connected to a master server at 127.0.71.125:40265
I20260929 01:53:32.866786 19233 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:32.866819 18744 ts_manager.cc:194] Registered new tserver with Master: d76c59eb7518455a933f8048f8a0386a (127.0.71.67:44367)
I20260929 01:53:32.867267 19233 heartbeater.cc:507] Master 127.0.71.125:40265 requested a full tablet report, sending...
I20260929 01:53:32.867611 18565 ts_manager.cc:194] Registered new tserver with Master: d76c59eb7518455a933f8048f8a0386a (127.0.71.67:44367)
I20260929 01:53:32.868429 18654 ts_manager.cc:194] Registered new tserver with Master: d76c59eb7518455a933f8048f8a0386a (127.0.71.67:44367)
I20260929 01:53:32.868794 18565 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.67@KRBTEST.COM'} at 127.0.71.67:60155
I20260929 01:53:32.878305 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:53:32.885363 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:53:32.885522 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 18533
I20260929 01:53:32.886416 18595 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:53:32.998317 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 18533
W20260929 01:53:33.020577 18729 connection.cc:573] server connection from 127.0.0.1:42586 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:53:33.021472 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:43705
--webserver_interface=127.0.71.126
--webserver_port=34189
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:44631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:33.275637 19090 heartbeater.cc:499] Master 127.0.71.126:43705 was elected leader, sending a full tablet report...
W20260929 01:53:33.277187 19090 heartbeater.cc:646] Failed to heartbeat to 127.0.71.126:43705 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.71.126:43705: connect: Connection refused (error 111)
W20260929 01:53:33.365861 19242 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:33.366240 19242 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:33.366355 19242 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:33.376677 19242 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:53:33.376807 19242 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:33.376914 19242 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:53:33.376981 19242 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:53:33.391240 19242 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:44631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:43705
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=34189
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:33.392819 19242 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:33.394080 19242 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:33.402354 19248 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:33.402405 19249 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:33.402695 19251 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:33.403434 19242 server_base.cc:1061] running on GCE node
I20260929 01:53:33.404275 19242 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:33.405737 19242 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:33.406934 19242 hybrid_clock.cc:648] HybridClock initialized: now 1790646813406884 us; error 51 us; skew 500 ppm
I20260929 01:53:33.407279 19242 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:33 dist-test-slave-vz23 krb5kdc[18517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:33.410959 19242 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:53:33.412735 19242 webserver.cc:533] Webserver started at http://127.0.71.126:34189/ using document root <none> and password file <none>
I20260929 01:53:33.413406 19242 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:33.413497 19242 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:33.419425 19242 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 01:53:33.422327 19258 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:33.423691 19242 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:53:33.423826 19242 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "0419fca067cd4c31ad19f304709d4d94"
format_stamp: "Formatted at 2026-09-29 01:53:20 on dist-test-slave-vz23"
I20260929 01:53:33.424589 19242 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:33.454632 19242 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:33.458971 19242 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:33.459192 19242 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:33.480039 19242 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:43705
I20260929 01:53:33.480077 19309 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:43705 every 8 connection(s)
I20260929 01:53:33.481696 19242 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:53:33.482496 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 19242
I20260929 01:53:33.482860 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 18810
I20260929 01:53:33.489339 19310 sys_catalog.cc:263] Verifying existing consensus state
Sep 29 01:53:33 dist-test-slave-vz23 krb5kdc[18517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646808, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 01:53:33.493021 19310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: Bootstrap starting.
I20260929 01:53:33.500959 18941 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:53:33.511759 19310 log.cc:826] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: Log is configured to *not* fsync() on all Append() calls
I20260929 01:53:33.521433 19310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:53:33.522048 19310 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: Bootstrap complete.
I20260929 01:53:33.529488 19310 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } }
I20260929 01:53:33.530339 19310 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0419fca067cd4c31ad19f304709d4d94, State: Initialized, Role: FOLLOWER
I20260929 01:53:33.531180 19310 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } }
I20260929 01:53:33.532027 19316 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:33.532327 19316 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:33.533140 19310 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:53:33.552214 19327 catalog_manager.cc:1308] Loaded cluster ID: 1e3f6d40824c4de3a0f91c80f6947064
I20260929 01:53:33.552464 19327 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: loading cluster ID for follower catalog manager: success
I20260929 01:53:33.556891 19327 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: acquiring CA information for follower catalog manager: success
I20260929 01:53:33.559676 19327 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Sep 29 01:53:33 dist-test-slave-vz23 krb5kdc[18517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646813, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:53:33.602174 19242 master_runner.cc:264] No leader master found from master 0419fca067cd4c31ad19f304709d4d94
I20260929 01:53:33.602329 19242 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:53:33.613281 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 18810
I20260929 01:53:33.636286 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:34225
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--webserver_port=41407
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:44631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:33.870875 19232 heartbeater.cc:499] Master 127.0.71.126:43705 was elected leader, sending a full tablet report...
I20260929 01:53:33.882009 19275 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d76c59eb7518455a933f8048f8a0386a" instance_seqno: 1790646812803047) as {username='kudu', principal='kudu/127.0.71.67@KRBTEST.COM'} at 127.0.71.67:55353; Asking this server to re-register.
I20260929 01:53:33.882965 19232 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:33.884678 19275 ts_manager.cc:194] Registered new tserver with Master: d76c59eb7518455a933f8048f8a0386a (127.0.71.67:44367)
W20260929 01:53:33.986927 19333 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:33.987306 19333 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:33.987471 19333 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:33.995867 19333 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:33.996147 19333 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:53:34.007601 19333 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:44631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:34225
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=41407
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:34.009443 19333 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:34.010776 19333 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:34.020856 19340 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:34.020929 19342 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:34.020972 19339 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:34.022003 19333 server_base.cc:1061] running on GCE node
I20260929 01:53:34.022657 19333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:34.023610 19333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:34.024847 19333 hybrid_clock.cc:648] HybridClock initialized: now 1790646814024814 us; error 40 us; skew 500 ppm
I20260929 01:53:34.025177 19333 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:34 dist-test-slave-vz23 krb5kdc[18517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:34.028950 19333 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:53:34.030711 19333 webserver.cc:533] Webserver started at http://127.0.71.65:41407/ using document root <none> and password file <none>
I20260929 01:53:34.031399 19333 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:34.031490 19333 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:34.037236 19333 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 01:53:34.040189 19349 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:34.041603 19333 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:53:34.041769 19333 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "2801d4e8cd0e4abd87fb76207115b8de"
format_stamp: "Formatted at 2026-09-29 01:53:22 on dist-test-slave-vz23"
I20260929 01:53:34.042476 19333 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:34.076093 19333 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:34.080332 19333 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:34.080642 19333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:34.081643 19333 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:34.083245 19333 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:34.083343 19333 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:34.083421 19333 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:34.083475 19333 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:34.119401 19333 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:34225
I20260929 01:53:34.119489 19461 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:34225 every 8 connection(s)
I20260929 01:53:34.121438 19333 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:53:34.122915 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 19333
Sep 29 01:53:34 dist-test-slave-vz23 krb5kdc[18517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:53:34 dist-test-slave-vz23 krb5kdc[18517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:53:34.160370 19464 heartbeater.cc:344] Connected to a master server at 127.0.71.125:40265
I20260929 01:53:34.160940 19464 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:34.162142 19464 heartbeater.cc:507] Master 127.0.71.125:40265 requested a full tablet report, sending...
I20260929 01:53:34.160470 19463 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43705
I20260929 01:53:34.163113 19463 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:34.163579 19463 heartbeater.cc:507] Master 127.0.71.126:43705 requested a full tablet report, sending...
I20260929 01:53:34.164558 18654 ts_manager.cc:194] Re-registered known tserver with Master: 2801d4e8cd0e4abd87fb76207115b8de (127.0.71.65:34225)
I20260929 01:53:34.164740 19275 ts_manager.cc:194] Registered new tserver with Master: 2801d4e8cd0e4abd87fb76207115b8de (127.0.71.65:34225)
Sep 29 01:53:34 dist-test-slave-vz23 krb5kdc[18517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:53:34.176548 19462 heartbeater.cc:344] Connected to a master server at 127.0.71.124:34639
I20260929 01:53:34.176728 19462 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:34.177440 19462 heartbeater.cc:507] Master 127.0.71.124:34639 requested a full tablet report, sending...
I20260929 01:53:34.178395 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 19333
I20260929 01:53:34.178913 18744 ts_manager.cc:194] Re-registered known tserver with Master: 2801d4e8cd0e4abd87fb76207115b8de (127.0.71.65:34225)
W20260929 01:53:34.179569 19474 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:34.171300 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:51062 (local address 127.0.71.65:34225)
0929 01:53:34.176381 (+ 5081us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:34.176388 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:34.176415 (+ 27us) server_negotiation.cc:408] Connection header received
0929 01:53:34.176535 (+ 120us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:53:34.176541 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:53:34.176965 (+ 424us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:53:34.177131 (+ 166us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:53:34.178225 (+ 1094us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51062: BlockingRecv error: recv got EOF from 127.0.0.1:51062 (error 108)
Metrics: {"server-negotiator.queue_time_us":4962,"thread_start_us":4773,"threads_started":1}
W20260929 01:53:34.181075 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 19333: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:34225: server requires authentication, but client does not have Kerberos credentials available
W20260929 01:53:34.181124 19474 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:34.179094 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:51070 (local address 127.0.71.65:34225)
0929 01:53:34.179996 (+ 902us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:34.180000 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:34.180018 (+ 18us) server_negotiation.cc:408] Connection header received
0929 01:53:34.180052 (+ 34us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:53:34.180055 (+ 3us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:53:34.180099 (+ 44us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:53:34.180206 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:53:34.180948 (+ 742us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51070: BlockingRecv error: recv got EOF from 127.0.0.1:51070 (error 108)
Metrics: {"server-negotiator.queue_time_us":839,"thread_start_us":1388,"threads_started":1}
I20260929 01:53:34.181191 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 19333
I20260929 01:53:34.198753 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:34225
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--webserver_port=41407
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:44631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:34.230798 19477 raft_consensus.cc:493] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0419fca067cd4c31ad19f304709d4d94)
I20260929 01:53:34.231060 19477 raft_consensus.cc:515] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } }
I20260929 01:53:34.232914 19477 leader_election.cc:290] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0419fca067cd4c31ad19f304709d4d94 (127.0.71.126:43705), 8c7ddf0bc0cb4650ae96cbe167b2e092 (127.0.71.124:34639)
I20260929 01:53:34.241044 19285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "222b3cc4d7c0489c8760afddb2484746" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0419fca067cd4c31ad19f304709d4d94" is_pre_election: true
I20260929 01:53:34.241591 19285 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 222b3cc4d7c0489c8760afddb2484746 in term 2.
I20260929 01:53:34.242175 18641 leader_election.cc:304] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0419fca067cd4c31ad19f304709d4d94, 222b3cc4d7c0489c8760afddb2484746; no voters:
I20260929 01:53:34.242574 19477 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 01:53:34.242699 19477 raft_consensus.cc:493] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 0419fca067cd4c31ad19f304709d4d94)
I20260929 01:53:34.242799 19477 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:53:34.243263 18754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "222b3cc4d7c0489c8760afddb2484746" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" is_pre_election: true
I20260929 01:53:34.245357 19477 raft_consensus.cc:515] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } }
I20260929 01:53:34.246016 19477 leader_election.cc:290] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [CANDIDATE]: Term 3 election: Requested vote from peers 0419fca067cd4c31ad19f304709d4d94 (127.0.71.126:43705), 8c7ddf0bc0cb4650ae96cbe167b2e092 (127.0.71.124:34639)
I20260929 01:53:34.246454 19285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "222b3cc4d7c0489c8760afddb2484746" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0419fca067cd4c31ad19f304709d4d94"
I20260929 01:53:34.246606 18754 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "222b3cc4d7c0489c8760afddb2484746" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092"
I20260929 01:53:34.246670 19285 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:53:34.250322 19285 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 222b3cc4d7c0489c8760afddb2484746 in term 3.
I20260929 01:53:34.250808 18641 leader_election.cc:304] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 0419fca067cd4c31ad19f304709d4d94, 222b3cc4d7c0489c8760afddb2484746; no voters: 8c7ddf0bc0cb4650ae96cbe167b2e092
I20260929 01:53:34.251132 19477 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 01:53:34.251703 19477 raft_consensus.cc:697] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [term 3 LEADER]: Becoming Leader. State: Replica: 222b3cc4d7c0489c8760afddb2484746, State: Running, Role: LEADER
I20260929 01:53:34.252108 19477 consensus_queue.cc:237] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } }
I20260929 01:53:34.254043 19480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 222b3cc4d7c0489c8760afddb2484746. Latest consensus state: current_term: 3 leader_uuid: "222b3cc4d7c0489c8760afddb2484746" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:34.254462 19480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [sys.catalog]: This master's current role is: LEADER
I20260929 01:53:34.255226 19482 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:53:34.257287 19482 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:53:34.258064 19482 catalog_manager.cc:1308] Loaded cluster ID: 1e3f6d40824c4de3a0f91c80f6947064
I20260929 01:53:34.258163 19482 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:53:34.259238 19482 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:53:34.260036 19482 catalog_manager.cc:6230] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746: Loaded TSK: 0
I20260929 01:53:34.260869 19482 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:53:34.284198 19090 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43705
I20260929 01:53:34.284518 19090 heartbeater.cc:499] Master 127.0.71.126:43705 was elected leader, sending a full tablet report...
I20260929 01:53:34.285593 19275 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d711f64efd3742f9919b75f0c4353415" instance_seqno: 1790646812208762) as {username='kudu', principal='kudu/127.0.71.66@KRBTEST.COM'} at 127.0.71.66:45325; Asking this server to re-register.
I20260929 01:53:34.286451 19090 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:34.287894 19275 ts_manager.cc:194] Registered new tserver with Master: d711f64efd3742f9919b75f0c4353415 (127.0.71.66:33363)
I20260929 01:53:34.499377 19484 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 0419fca067cd4c31ad19f304709d4d94)
I20260929 01:53:34.499589 19484 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } }
I20260929 01:53:34.501492 19484 leader_election.cc:290] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 0419fca067cd4c31ad19f304709d4d94 (127.0.71.126:43705), 222b3cc4d7c0489c8760afddb2484746 (127.0.71.125:40265)
I20260929 01:53:34.502926 18664 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "222b3cc4d7c0489c8760afddb2484746" is_pre_election: true
I20260929 01:53:34.508589 19285 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "0419fca067cd4c31ad19f304709d4d94" is_pre_election: true
I20260929 01:53:34.508994 19285 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8c7ddf0bc0cb4650ae96cbe167b2e092 in current term 3: Already voted for candidate 222b3cc4d7c0489c8760afddb2484746 in this term.
I20260929 01:53:34.509644 18731 leader_election.cc:304] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8c7ddf0bc0cb4650ae96cbe167b2e092; no voters: 0419fca067cd4c31ad19f304709d4d94, 222b3cc4d7c0489c8760afddb2484746
I20260929 01:53:34.510064 19484 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:53:34.512159 19484 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260929 01:53:34.553086 19476 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:34.553377 19476 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:34.553476 19476 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:53:34.553535 19476 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:34.562119 19476 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:34.562323 19476 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:53:34.573197 19476 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:44631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:34225
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=41407
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:34.574695 19476 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:34.575945 19476 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:34.585147 19490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:34.585451 19493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:34.585757 19491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:34.586336 19476 server_base.cc:1061] running on GCE node
I20260929 01:53:34.587049 19476 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:34.588027 19476 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:34.589254 19476 hybrid_clock.cc:648] HybridClock initialized: now 1790646814589194 us; error 32 us; skew 500 ppm
I20260929 01:53:34.589516 19476 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:34 dist-test-slave-vz23 krb5kdc[18517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646814, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:34.593161 19476 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:53:34.594787 19476 webserver.cc:533] Webserver started at http://127.0.71.65:41407/ using document root <none> and password file <none>
I20260929 01:53:34.595506 19476 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:34.595613 19476 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:34.601397 19476 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:53:34.604368 19500 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:34.605724 19476 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:53:34.605861 19476 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "2801d4e8cd0e4abd87fb76207115b8de"
format_stamp: "Formatted at 2026-09-29 01:53:22 on dist-test-slave-vz23"
I20260929 01:53:34.606598 19476 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:34.618165 19476 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:34.627084 19476 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:34.627368 19476 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:34.628350 19476 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:34.630205 19476 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:34.630292 19476 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:34.630461 19476 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:34.630558 19476 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:53:34.630955 19476 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:53:34.631022 19476 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:53:34.631124 19476 fault_injection.cc:59] injecting fault for kudu (pid 19476): FLAGS_fault_before_start (process will exit)
I20260929 01:53:34.650961 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:34225
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--webserver_port=41407
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:44631
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:34.730908 18754 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [term 3 FOLLOWER]: Refusing update from remote peer 222b3cc4d7c0489c8760afddb2484746: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 01:53:34.731703 19480 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:53:34.733595 19484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 222b3cc4d7c0489c8760afddb2484746. Latest consensus state: current_term: 3 leader_uuid: "222b3cc4d7c0489c8760afddb2484746" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:34.733922 19484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:34.735745 19480 sys_catalog.cc:455] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "222b3cc4d7c0489c8760afddb2484746" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:34.736043 19480 sys_catalog.cc:458] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [sys.catalog]: This master's current role is: LEADER
I20260929 01:53:34.737251 19285 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [term 3 FOLLOWER]: Refusing update from remote peer 222b3cc4d7c0489c8760afddb2484746: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 01:53:34.738091 19511 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [LEADER]: Connected to new peer: Peer: permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:53:34.738304 19484 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "222b3cc4d7c0489c8760afddb2484746" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:34.738518 19484 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8c7ddf0bc0cb4650ae96cbe167b2e092 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:34.742663 19516 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "222b3cc4d7c0489c8760afddb2484746" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:34.743248 19516 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:34.743420 19513 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 222b3cc4d7c0489c8760afddb2484746. Latest consensus state: current_term: 3 leader_uuid: "222b3cc4d7c0489c8760afddb2484746" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:34.743685 19513 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0419fca067cd4c31ad19f304709d4d94 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:53:34.993613 19508 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:34.993938 19508 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:34.994120 19508 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:35.003340 19508 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:35.003602 19508 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:53:35.018460 19508 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:44631
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:34225
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=41407
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:43705,127.0.71.125:40265,127.0.71.124:34639
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:35.020077 19508 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:35.021528 19508 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:35.030721 19521 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:35.030769 19522 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:35.031813 19524 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:35.032763 19508 server_base.cc:1061] running on GCE node
I20260929 01:53:35.033360 19508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:35.034443 19508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:35.035702 19508 hybrid_clock.cc:648] HybridClock initialized: now 1790646815035642 us; error 52 us; skew 500 ppm
I20260929 01:53:35.036017 19508 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:35 dist-test-slave-vz23 krb5kdc[18517](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:35.044428 19508 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:53:35.046247 19508 webserver.cc:533] Webserver started at http://127.0.71.65:41407/ using document root <none> and password file <none>
I20260929 01:53:35.047006 19508 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:35.047148 19508 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:35.052832 19508 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 01:53:35.058605 19531 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:35.060578 19508 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260929 01:53:35.060791 19508 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "2801d4e8cd0e4abd87fb76207115b8de"
format_stamp: "Formatted at 2026-09-29 01:53:22 on dist-test-slave-vz23"
I20260929 01:53:35.061884 19508 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:35.083863 19508 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:35.087558 19508 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:35.087940 19508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:35.089037 19508 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:35.090579 19508 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:35.090713 19508 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:35.090840 19508 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:35.090931 19508 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:35.133841 19508 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:34225
I20260929 01:53:35.133888 19643 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:34225 every 8 connection(s)
I20260929 01:53:35.135736 19508 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:53:35.145942 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 19508
I20260929 01:53:35.146257 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 19508
Sep 29 01:53:35 dist-test-slave-vz23 krb5kdc[18517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:53:35 dist-test-slave-vz23 krb5kdc[18517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:53:35 dist-test-slave-vz23 krb5kdc[18517](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646815, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
W20260929 01:53:35.167258 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 19508: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:34225: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:53:35.167413 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 19508
I20260929 01:53:35.175763 19511 sys_catalog.cc:455] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "222b3cc4d7c0489c8760afddb2484746" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "0419fca067cd4c31ad19f304709d4d94" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43705 } } peers { permanent_uuid: "222b3cc4d7c0489c8760afddb2484746" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40265 } } peers { permanent_uuid: "8c7ddf0bc0cb4650ae96cbe167b2e092" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34639 } } }
I20260929 01:53:35.176043 19511 sys_catalog.cc:458] T 00000000000000000000000000000000 P 222b3cc4d7c0489c8760afddb2484746 [sys.catalog]: This master's current role is: LEADER
W20260929 01:53:35.186147 19653 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:35.143420 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:39301 (local address 127.0.71.126:43705)
0929 01:53:35.143907 (+ 487us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:35.143916 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:35.144800 (+ 884us) server_negotiation.cc:408] Connection header received
0929 01:53:35.144902 (+ 102us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:53:35.144907 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:53:35.145039 (+ 132us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:53:35.145152 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:53:35.148990 (+ 3838us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:53:35.150482 (+ 1492us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:53:35.153464 (+ 2982us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:53:35.153959 (+ 495us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0929 01:53:35.159570 (+ 5611us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0929 01:53:35.159647 (+ 77us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0929 01:53:35.159656 (+ 9us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0929 01:53:35.159706 (+ 50us) server_negotiation.cc:923] calling sasl_server_start()
0929 01:53:35.164554 (+ 4848us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0929 01:53:35.165955 (+ 1401us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0929 01:53:35.165964 (+ 9us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0929 01:53:35.165969 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0929 01:53:35.166081 (+ 112us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0929 01:53:35.166734 (+ 653us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0929 01:53:35.166742 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0929 01:53:35.166747 (+ 5us) server_negotiation.cc:968] calling sasl_server_step()
0929 01:53:35.167188 (+ 441us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0929 01:53:35.168602 (+ 1414us) server_negotiation.cc:1036] Waiting for connection context
0929 01:53:35.185296 (+ 16694us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:39301: BlockingRecv error: recv got EOF from 127.0.71.65:39301 (error 108)
Metrics: {"server-negotiator.queue_time_us":366,"thread_start_us":271,"threads_started":1}
W20260929 01:53:35.186203 19651 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:35.141594 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:59393 (local address 127.0.71.124:34639)
0929 01:53:35.142493 (+ 899us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:35.142499 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:35.142519 (+ 20us) server_negotiation.cc:408] Connection header received
0929 01:53:35.144591 (+ 2072us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:53:35.144598 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:53:35.144696 (+ 98us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:53:35.144821 (+ 125us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:53:35.149166 (+ 4345us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:53:35.150978 (+ 1812us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:53:35.154429 (+ 3451us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:53:35.154785 (+ 356us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0929 01:53:35.166383 (+ 11598us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0929 01:53:35.166424 (+ 41us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0929 01:53:35.166430 (+ 6us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0929 01:53:35.166468 (+ 38us) server_negotiation.cc:923] calling sasl_server_start()
0929 01:53:35.179595 (+ 13127us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0929 01:53:35.185350 (+ 5755us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:59393: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":778,"thread_start_us":1708,"threads_started":1}
W20260929 01:53:35.186233 19648 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:35.139249 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:46793 (local address 127.0.71.125:40265)
0929 01:53:35.139848 (+ 599us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:35.139858 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:35.141485 (+ 1627us) server_negotiation.cc:408] Connection header received
0929 01:53:35.144644 (+ 3159us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:53:35.144650 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:53:35.144748 (+ 98us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:53:35.144874 (+ 126us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:53:35.148020 (+ 3146us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:53:35.149527 (+ 1507us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:53:35.151161 (+ 1634us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:53:35.151532 (+ 371us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0929 01:53:35.159900 (+ 8368us) server_negotiation.cc:366] Received SASL_INITIATE NegotiatePB request
0929 01:53:35.159929 (+ 29us) server_negotiation.cc:882] Received SASL_INITIATE request from client
0929 01:53:35.159933 (+ 4us) server_negotiation.cc:893] Client requested to use mechanism: GSSAPI
0929 01:53:35.159963 (+ 30us) server_negotiation.cc:923] calling sasl_server_start()
0929 01:53:35.164937 (+ 4974us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0929 01:53:35.166044 (+ 1107us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0929 01:53:35.166051 (+ 7us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0929 01:53:35.166055 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0929 01:53:35.166152 (+ 97us) server_negotiation.cc:378] Sending SASL_CHALLENGE NegotiatePB response
0929 01:53:35.167037 (+ 885us) server_negotiation.cc:366] Received SASL_RESPONSE NegotiatePB request
0929 01:53:35.167045 (+ 8us) server_negotiation.cc:957] Received SASL_RESPONSE request from client
0929 01:53:35.167049 (+ 4us) server_negotiation.cc:968] calling sasl_server_step()
0929 01:53:35.169105 (+ 2056us) server_negotiation.cc:378] Sending SASL_SUCCESS NegotiatePB response
0929 01:53:35.169201 (+ 96us) server_negotiation.cc:1036] Waiting for connection context
0929 01:53:35.185341 (+ 16140us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:46793: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":444,"thread_start_us":286,"threads_started":1}
I20260929 01:53:35.186888 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 18960
W20260929 01:53:35.189733 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 18960: Not authorized: Client connection negotiation failed: client connection to 127.0.71.66:33363: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:53:35.189913 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 18960
I20260929 01:53:35.205910 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 19103
W20260929 01:53:35.208726 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 19103: Not authorized: Client connection negotiation failed: client connection to 127.0.71.67:44367: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:53:35.208900 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 19103
I20260929 01:53:35.224808 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 19242
W20260929 01:53:35.227075 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 19242: Not authorized: Client connection negotiation failed: client connection to 127.0.71.126:43705: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:53:35.227226 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 19242
I20260929 01:53:35.248003 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 18622
I20260929 01:53:35.249140 18684 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:53:35.362298 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 18622
I20260929 01:53:35.386541 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 18712
I20260929 01:53:35.387802 18774 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:53:35.500310 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 18712
2026-09-29T01:53:35Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestBasicOperation/7 (18048 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/8
Loading random data
Initializing database '/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:53:35 dist-test-slave-vz23 krb5kdc[19663](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:53:35 dist-test-slave-vz23 krb5kdc[19663](info): set up 2 sockets
Sep 29 01:53:35 dist-test-slave-vz23 krb5kdc[19663](info): commencing operation
krb5kdc: starting...
W20260929 01:53:37.601637 285 mini_kdc.cc:121] Time spent starting KDC: real 2.056s user 0.003s sys 0.018s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 01:53:37 dist-test-slave-vz23 krb5kdc[19663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:53:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:53:37Z Disabled control of system clock
2026-09-29T01:53:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:53:37Z Disabled control of system clock
2026-09-29T01:53:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:53:37Z Disabled control of system clock
2026-09-29T01:53:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:53:37Z Disabled control of system clock
2026-09-29T01:53:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:53:37Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:37.970077 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:43657
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:40553,127.0.71.85:38001,127.0.71.86:41825,127.0.71.87:42223,127.0.71.88:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:53:38.305474 19687 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:38.305872 19687 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:38.306027 19687 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:38.314793 19687 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:53:38.314956 19687 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:38.315068 19687 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:53:38.315157 19687 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:53:38.326171 19687 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:40553,127.0.71.85:38001,127.0.71.86:41825,127.0.71.87:42223,127.0.71.88:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:43657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:38.327833 19687 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:38.329335 19687 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:38.337527 19692 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:38.337647 19693 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:38.338163 19695 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:38.338807 19687 server_base.cc:1061] running on GCE node
I20260929 01:53:38.339581 19687 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:38.341286 19687 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:38.342494 19687 hybrid_clock.cc:648] HybridClock initialized: now 1790646818342355 us; error 108 us; skew 500 ppm
I20260929 01:53:38.342855 19687 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:38 dist-test-slave-vz23 krb5kdc[19663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:38.346130 19687 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:53:38.347895 19687 webserver.cc:533] Webserver started at http://127.0.71.126:33127/ using document root <none> and password file <none>
I20260929 01:53:38.348737 19687 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:38.348871 19687 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:38.349229 19687 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:53:38.351680 19687 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "ce7ce17b42f04948b67de74b41285e11"
format_stamp: "Formatted at 2026-09-29 01:53:38 on dist-test-slave-vz23"
I20260929 01:53:38.352527 19687 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "ce7ce17b42f04948b67de74b41285e11"
format_stamp: "Formatted at 2026-09-29 01:53:38 on dist-test-slave-vz23"
I20260929 01:53:38.357812 19687 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.001s
I20260929 01:53:38.361244 19702 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:38.362649 19687 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:53:38.362825 19687 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "ce7ce17b42f04948b67de74b41285e11"
format_stamp: "Formatted at 2026-09-29 01:53:38 on dist-test-slave-vz23"
I20260929 01:53:38.363010 19687 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:38.382571 19687 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:38.386348 19687 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:38.386638 19687 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:38.407281 19687 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:43657
I20260929 01:53:38.407316 19753 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:43657 every 8 connection(s)
I20260929 01:53:38.408905 19687 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:53:38.411569 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 19687
I20260929 01:53:38.411773 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:53:38.414508 19754 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:53:38.422317 19754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.71.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.125@KRBTEST.COM" created.
Sep 29 01:53:38 dist-test-slave-vz23 krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:53:38.446565 19754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:53:38.447779 19704 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.125:36483: connect: Connection refused (error 111)
W20260929 01:53:38.449384 19754 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:36483: Network error: Client connection negotiation failed: client connection to 127.0.71.125:36483: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.125@KRBTEST.COM" created.
I20260929 01:53:38.498733 19754 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } attempt: 1
W20260929 01:53:38.500929 19754 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:36483: Network error: Client connection negotiation failed: client connection to 127.0.71.125:36483: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:38.510022 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:36483
--webserver_interface=127.0.71.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:40553,127.0.71.85:38001,127.0.71.86:41825,127.0.71.87:42223,127.0.71.88:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:38.569201 19754 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } attempt: 2
W20260929 01:53:38.571302 19754 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:36483: Network error: Client connection negotiation failed: client connection to 127.0.71.125:36483: connect: Connection refused (error 111)
I20260929 01:53:38.662546 19754 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } attempt: 3
W20260929 01:53:38.664690 19754 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:36483: Network error: Client connection negotiation failed: client connection to 127.0.71.125:36483: connect: Connection refused (error 111)
I20260929 01:53:38.807956 19754 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } attempt: 4
W20260929 01:53:38.810061 19754 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:36483: Network error: Client connection negotiation failed: client connection to 127.0.71.125:36483: connect: Connection refused (error 111)
W20260929 01:53:38.843042 19763 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:38.843425 19763 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:38.843585 19763 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:38.852092 19763 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:53:38.852314 19763 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:38.852468 19763 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:53:38.852586 19763 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:53:38.863500 19763 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:40553,127.0.71.85:38001,127.0.71.86:41825,127.0.71.87:42223,127.0.71.88:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:36483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:38.865338 19763 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:38.866844 19763 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:38.875555 19768 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:38.875555 19769 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:38.876691 19771 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:38.877283 19763 server_base.cc:1061] running on GCE node
I20260929 01:53:38.878074 19763 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:38.879666 19763 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:38.880985 19763 hybrid_clock.cc:648] HybridClock initialized: now 1790646818880807 us; error 144 us; skew 500 ppm
I20260929 01:53:38.881336 19763 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:38 dist-test-slave-vz23 krb5kdc[19663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:38.884418 19763 init.cc:377] Logged in from keytab as kudu/127.0.71.125@KRBTEST.COM (short username kudu)
I20260929 01:53:38.886009 19763 webserver.cc:533] Webserver started at http://127.0.71.125:40085/ using document root <none> and password file <none>
I20260929 01:53:38.886715 19763 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:38.886852 19763 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:38.887226 19763 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:53:38.889688 19763 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/data/instance:
uuid: "444f97234e6d4c20be102cb7c0628408"
format_stamp: "Formatted at 2026-09-29 01:53:38 on dist-test-slave-vz23"
I20260929 01:53:38.890547 19763 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/wal/instance:
uuid: "444f97234e6d4c20be102cb7c0628408"
format_stamp: "Formatted at 2026-09-29 01:53:38 on dist-test-slave-vz23"
I20260929 01:53:38.895845 19763 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 01:53:38.899221 19778 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:38.900666 19763 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:53:38.900846 19763 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "444f97234e6d4c20be102cb7c0628408"
format_stamp: "Formatted at 2026-09-29 01:53:38 on dist-test-slave-vz23"
I20260929 01:53:38.901037 19763 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:38.918176 19763 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:38.921902 19763 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:38.922188 19763 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:38.942656 19763 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:36483
I20260929 01:53:38.942706 19829 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:36483 every 8 connection(s)
I20260929 01:53:38.944216 19763 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:53:38.948783 19830 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:53:38.951491 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 19763
I20260929 01:53:38.951697 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:53:38.954840 19830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:53:38 dist-test-slave-vz23 krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.124@KRBTEST.COM" created.
I20260929 01:53:38.977294 19830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:53:38 dist-test-slave-vz23 krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:53:38.994752 19830 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:53:38.995903 19782 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43011: connect: Connection refused (error 111)
W20260929 01:53:38.997620 19830 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43011: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43011: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.124@KRBTEST.COM" created.
I20260929 01:53:39.047009 19830 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } attempt: 1
W20260929 01:53:39.049427 19830 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43011: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43011: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:39.051869 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:43011
--webserver_interface=127.0.71.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:40553,127.0.71.85:38001,127.0.71.86:41825,127.0.71.87:42223,127.0.71.88:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:39.109324 19754 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } attempt: 5
Sep 29 01:53:39 dist-test-slave-vz23 krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:53:39.117646 19830 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } attempt: 2
W20260929 01:53:39.119872 19830 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43011: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43011: connect: Connection refused (error 111)
I20260929 01:53:39.125352 19754 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:53:39.127727 19754 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43011: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43011: connect: Connection refused (error 111)
I20260929 01:53:39.178966 19754 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } attempt: 1
W20260929 01:53:39.181109 19754 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43011: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43011: connect: Connection refused (error 111)
I20260929 01:53:39.211126 19830 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } attempt: 3
W20260929 01:53:39.213471 19830 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43011: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43011: connect: Connection refused (error 111)
I20260929 01:53:39.249341 19754 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } attempt: 2
W20260929 01:53:39.252007 19754 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43011: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43011: connect: Connection refused (error 111)
I20260929 01:53:39.356730 19830 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } attempt: 4
I20260929 01:53:39.358260 19754 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } attempt: 3
W20260929 01:53:39.359429 19830 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43011: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43011: connect: Connection refused (error 111)
W20260929 01:53:39.360998 19754 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43011: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43011: connect: Connection refused (error 111)
W20260929 01:53:39.401733 19840 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:39.402156 19840 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:39.402365 19840 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:39.415594 19840 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:53:39.415771 19840 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:39.415899 19840 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:53:39.415989 19840 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:53:39.429560 19840 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:40553,127.0.71.85:38001,127.0.71.86:41825,127.0.71.87:42223,127.0.71.88:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:43011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:39.431275 19840 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:39.432737 19840 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:39.441015 19846 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:39.440970 19845 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:39.441850 19848 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:39.442478 19840 server_base.cc:1061] running on GCE node
I20260929 01:53:39.443336 19840 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:39.445029 19840 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:39.446228 19840 hybrid_clock.cc:648] HybridClock initialized: now 1790646819446052 us; error 145 us; skew 500 ppm
I20260929 01:53:39.446601 19840 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:39 dist-test-slave-vz23 krb5kdc[19663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:39.449803 19840 init.cc:377] Logged in from keytab as kudu/127.0.71.124@KRBTEST.COM (short username kudu)
I20260929 01:53:39.451325 19840 webserver.cc:533] Webserver started at http://127.0.71.124:46541/ using document root <none> and password file <none>
I20260929 01:53:39.452028 19840 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:39.452203 19840 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:39.452581 19840 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:53:39.454932 19840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/data/instance:
uuid: "8cf5932291314506937aeaa8e67f3dff"
format_stamp: "Formatted at 2026-09-29 01:53:39 on dist-test-slave-vz23"
I20260929 01:53:39.455652 19840 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/wal/instance:
uuid: "8cf5932291314506937aeaa8e67f3dff"
format_stamp: "Formatted at 2026-09-29 01:53:39 on dist-test-slave-vz23"
I20260929 01:53:39.461383 19840 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.003s
I20260929 01:53:39.465086 19855 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:39.466404 19840 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:53:39.466586 19840 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "8cf5932291314506937aeaa8e67f3dff"
format_stamp: "Formatted at 2026-09-29 01:53:39 on dist-test-slave-vz23"
I20260929 01:53:39.466766 19840 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:39.480170 19840 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:39.483783 19840 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:39.484063 19840 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:39.504736 19840 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:43011
I20260929 01:53:39.504787 19906 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:43011 every 8 connection(s)
I20260929 01:53:39.506310 19840 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:53:39.510592 19907 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:53:39.513528 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 19840
I20260929 01:53:39.513743 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:53:39.517920 19907 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:53:39 dist-test-slave-vz23 krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:53:39 dist-test-slave-vz23 krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:53:39.538269 19754 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } attempt: 4
Sep 29 01:53:39 dist-test-slave-vz23 krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:53:39.540568 19907 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:53:39 dist-test-slave-vz23 krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:53:39 dist-test-slave-vz23 krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:53:39.557345 19907 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:53:39 dist-test-slave-vz23 krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:53:39.562327 19754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: Bootstrap starting.
I20260929 01:53:39.566793 19754 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: Neither blocks nor log segments found. Creating new log.
Sep 29 01:53:39 dist-test-slave-vz23 krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646819, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:53:39.568197 19754 log.cc:826] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: Log is configured to *not* fsync() on all Append() calls
I20260929 01:53:39.570799 19754 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: No bootstrap required, opened a new log
I20260929 01:53:39.577086 19754 raft_consensus.cc:359] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } }
I20260929 01:53:39.577386 19754 raft_consensus.cc:385] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:53:39.577523 19754 raft_consensus.cc:740] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce7ce17b42f04948b67de74b41285e11, State: Initialized, Role: FOLLOWER
I20260929 01:53:39.578161 19754 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } }
I20260929 01:53:39.578522 19907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff: Bootstrap starting.
I20260929 01:53:39.579197 19917 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:39.579617 19917 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:39.580577 19754 sys_catalog.cc:565] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:53:39.582305 19907 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff: Neither blocks nor log segments found. Creating new log.
I20260929 01:53:39.583308 19907 log.cc:826] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff: Log is configured to *not* fsync() on all Append() calls
I20260929 01:53:39.585840 19907 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff: No bootstrap required, opened a new log
W20260929 01:53:39.593211 19929 catalog_manager.cc:1607] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:53:39.593429 19929 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:53:39.593432 19907 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } }
I20260929 01:53:39.593793 19907 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:53:39.593933 19907 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cf5932291314506937aeaa8e67f3dff, State: Initialized, Role: FOLLOWER
I20260929 01:53:39.594611 19907 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } }
I20260929 01:53:39.595582 19931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:39.595937 19931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:39.596774 19907 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:53:39.590786 19687 master_runner.cc:187] Error getting master registration for 127.0.71.125:36483: OK, instance_id { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" instance_seqno: 1790646818937084 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:53:39.597086 19687 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:53:39.607827 19942 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:53:39.608110 19942 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:53:39.609172 19840 master_runner.cc:187] Error getting master registration for 127.0.71.125:36483: OK, instance_id { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" instance_seqno: 1790646818937084 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:53:39.615351 19840 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:53:39.618726 19931 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:53:39.618993 19931 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } }
I20260929 01:53:39.620534 19931 leader_election.cc:290] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers ce7ce17b42f04948b67de74b41285e11 (127.0.71.126:43657), 444f97234e6d4c20be102cb7c0628408 (127.0.71.125:36483)
I20260929 01:53:39.621234 19729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8cf5932291314506937aeaa8e67f3dff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce7ce17b42f04948b67de74b41285e11" is_pre_election: true
I20260929 01:53:39.621791 19729 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cf5932291314506937aeaa8e67f3dff in term 0.
I20260929 01:53:39.622514 19856 leader_election.cc:304] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cf5932291314506937aeaa8e67f3dff, ce7ce17b42f04948b67de74b41285e11; no voters:
I20260929 01:53:39.623977 19931 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:53:39.624135 19931 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:53:39.624261 19931 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:53:39.627739 19931 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } }
I20260929 01:53:39.628885 19729 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8cf5932291314506937aeaa8e67f3dff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ce7ce17b42f04948b67de74b41285e11"
I20260929 01:53:39.628986 19931 leader_election.cc:290] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [CANDIDATE]: Term 1 election: Requested vote from peers ce7ce17b42f04948b67de74b41285e11 (127.0.71.126:43657), 444f97234e6d4c20be102cb7c0628408 (127.0.71.125:36483)
I20260929 01:53:39.629098 19729 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:53:39.632583 19729 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cf5932291314506937aeaa8e67f3dff in term 1.
I20260929 01:53:39.633105 19856 leader_election.cc:304] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cf5932291314506937aeaa8e67f3dff, ce7ce17b42f04948b67de74b41285e11; no voters:
I20260929 01:53:39.633510 19931 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [term 1 FOLLOWER]: Leader election won for term 1
W20260929 01:53:39.639442 19708 tablet.cc:2433] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:53:39.643695 19931 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [term 1 LEADER]: Becoming Leader. State: Replica: 8cf5932291314506937aeaa8e67f3dff, State: Running, Role: LEADER
I20260929 01:53:39.644250 19931 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } }
I20260929 01:53:39.649353 19944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cf5932291314506937aeaa8e67f3dff. Latest consensus state: current_term: 1 leader_uuid: "8cf5932291314506937aeaa8e67f3dff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:39.649703 19944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [sys.catalog]: This master's current role is: LEADER
I20260929 01:53:39.656710 19946 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:53:39.658666 19830 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } attempt: 5
I20260929 01:53:39.659603 19946 catalog_manager.cc:1533] Initializing Kudu cluster ID...
Sep 29 01:53:39 dist-test-slave-vz23 krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646818, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:53:39.678087 19729 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [term 1 FOLLOWER]: Refusing update from remote peer 8cf5932291314506937aeaa8e67f3dff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:53:39.680115 19944 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:53:39.707877 19917 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cf5932291314506937aeaa8e67f3dff. Latest consensus state: current_term: 1 leader_uuid: "8cf5932291314506937aeaa8e67f3dff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:39.708169 19917 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:39.712284 19830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408: Bootstrap starting.
I20260929 01:53:39.713719 19948 mvcc.cc:204] Tried to move back new op lower bound from 7334489373385252864 to 7334489373267767296. Current Snapshot: MvccSnapshot[applied={T|T < 7334489373385252864}]
I20260929 01:53:39.726019 19931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8cf5932291314506937aeaa8e67f3dff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:39.726323 19931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [sys.catalog]: This master's current role is: LEADER
I20260929 01:53:39.727622 19830 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408: Neither blocks nor log segments found. Creating new log.
I20260929 01:53:39.728241 19946 catalog_manager.cc:1396] Generated new cluster ID: 1d32bc027e374184a8e577eeec0bb9fd
I20260929 01:53:39.728348 19946 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:53:39.729223 19830 log.cc:826] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408: Log is configured to *not* fsync() on all Append() calls
I20260929 01:53:39.731572 19917 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8cf5932291314506937aeaa8e67f3dff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:39.731772 19917 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:39.739342 19830 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408: No bootstrap required, opened a new log
I20260929 01:53:39.751358 19830 raft_consensus.cc:359] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } }
I20260929 01:53:39.751807 19830 raft_consensus.cc:385] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:53:39.751984 19830 raft_consensus.cc:740] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 444f97234e6d4c20be102cb7c0628408, State: Initialized, Role: FOLLOWER
I20260929 01:53:39.752784 19830 consensus_queue.cc:260] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } }
I20260929 01:53:39.759819 19953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:39.762049 19953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:39.761909 19830 sys_catalog.cc:565] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:53:39.762296 19803 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:53:39.764582 19805 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8cf5932291314506937aeaa8e67f3dff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "444f97234e6d4c20be102cb7c0628408" is_pre_election: true
I20260929 01:53:39.765661 19804 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8cf5932291314506937aeaa8e67f3dff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "444f97234e6d4c20be102cb7c0628408"
I20260929 01:53:39.768265 19803 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [term 1 FOLLOWER]: Refusing update from remote peer 8cf5932291314506937aeaa8e67f3dff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:53:39.770586 19944 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [LEADER]: Connected to new peer: Peer: permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:53:39.807108 19953 sys_catalog.cc:455] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cf5932291314506937aeaa8e67f3dff. Latest consensus state: current_term: 1 leader_uuid: "8cf5932291314506937aeaa8e67f3dff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:39.807242 19958 sys_catalog.cc:455] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "8cf5932291314506937aeaa8e67f3dff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:39.807432 19953 sys_catalog.cc:458] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:39.807510 19958 sys_catalog.cc:458] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:39.821983 19931 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8cf5932291314506937aeaa8e67f3dff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:39.822398 19931 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [sys.catalog]: This master's current role is: LEADER
I20260929 01:53:39.831372 19946 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 01:53:39.833328 19946 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:53:39.864833 19946 catalog_manager.cc:6219] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff: Generated new TSK 0
I20260929 01:53:39.866251 19946 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:53:39.892681 19970 catalog_manager.cc:1308] Loaded cluster ID: 1d32bc027e374184a8e577eeec0bb9fd
I20260929 01:53:39.892802 19970 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408: loading cluster ID for follower catalog manager: success
I20260929 01:53:39.896996 19970 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408: acquiring CA information for follower catalog manager: success
I20260929 01:53:39.907863 19970 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:40.044519 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:40553,127.0.71.85:38001,127.0.71.86:41825,127.0.71.87:42223,127.0.71.88:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:53:40.483042 19976 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:40.483407 19976 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:40.483593 19976 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:40.492404 19976 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:40.492727 19976 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:53:40.503698 19976 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:40553,127.0.71.85:38001,127.0.71.86:41825,127.0.71.87:42223,127.0.71.88:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:40.505474 19976 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:40.506927 19976 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:40.516718 19981 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:40.518096 19982 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:40.518936 19984 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:40.520198 19976 server_base.cc:1061] running on GCE node
I20260929 01:53:40.520857 19976 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:40.522069 19976 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:40.523468 19976 hybrid_clock.cc:648] HybridClock initialized: now 1790646820523294 us; error 134 us; skew 500 ppm
I20260929 01:53:40.523787 19976 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:40 dist-test-slave-vz23 krb5kdc[19663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:40.527324 19976 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:53:40.529098 19976 webserver.cc:533] Webserver started at http://127.0.71.65:39543/ using document root <none> and password file <none>
I20260929 01:53:40.529820 19976 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:40.529955 19976 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:40.530292 19976 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:53:40.532830 19976 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "ca2810190b0f4b078d7d671be7879084"
format_stamp: "Formatted at 2026-09-29 01:53:40 on dist-test-slave-vz23"
I20260929 01:53:40.533599 19976 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "ca2810190b0f4b078d7d671be7879084"
format_stamp: "Formatted at 2026-09-29 01:53:40 on dist-test-slave-vz23"
I20260929 01:53:40.538925 19976 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:53:40.542268 19991 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:40.543622 19976 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:53:40.543795 19976 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "ca2810190b0f4b078d7d671be7879084"
format_stamp: "Formatted at 2026-09-29 01:53:40 on dist-test-slave-vz23"
I20260929 01:53:40.543983 19976 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:40.559955 19976 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:40.564126 19976 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:40.564420 19976 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:40.565428 19976 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:40.566941 19976 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:40.567040 19976 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:40.567191 19976 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:40.567262 19976 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:40.595458 19929 catalog_manager.cc:1308] Loaded cluster ID: 1d32bc027e374184a8e577eeec0bb9fd
I20260929 01:53:40.595561 19929 catalog_manager.cc:1601] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: loading cluster ID for follower catalog manager: success
I20260929 01:53:40.599530 19929 catalog_manager.cc:1623] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: acquiring CA information for follower catalog manager: success
I20260929 01:53:40.602066 19929 catalog_manager.cc:1651] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:53:40.602932 19976 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:41705
I20260929 01:53:40.603017 20103 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:41705 every 8 connection(s)
I20260929 01:53:40.604434 19976 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:53:40.606294 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 19976
I20260929 01:53:40.606482 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
Sep 29 01:53:40 dist-test-slave-vz23 krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:53:40 dist-test-slave-vz23 krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:53:40 dist-test-slave-vz23 krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646820, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.66@KRBTEST.COM" created.
I20260929 01:53:40.663520 20104 heartbeater.cc:344] Connected to a master server at 127.0.71.124:43011
I20260929 01:53:40.663885 20104 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:40.665017 20104 heartbeater.cc:507] Master 127.0.71.124:43011 requested a full tablet report, sending...
I20260929 01:53:40.667547 19872 ts_manager.cc:194] Registered new tserver with Master: ca2810190b0f4b078d7d671be7879084 (127.0.71.65:41705)
I20260929 01:53:40.667776 20105 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43657
I20260929 01:53:40.667894 20105 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:40.668471 20105 heartbeater.cc:507] Master 127.0.71.126:43657 requested a full tablet report, sending...
I20260929 01:53:40.669497 19872 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:38145
I20260929 01:53:40.669852 20106 heartbeater.cc:344] Connected to a master server at 127.0.71.125:36483
I20260929 01:53:40.670001 20106 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:40.670536 20106 heartbeater.cc:507] Master 127.0.71.125:36483 requested a full tablet report, sending...
I20260929 01:53:40.672315 19719 ts_manager.cc:194] Registered new tserver with Master: ca2810190b0f4b078d7d671be7879084 (127.0.71.65:41705)
I20260929 01:53:40.673945 19795 ts_manager.cc:194] Registered new tserver with Master: ca2810190b0f4b078d7d671be7879084 (127.0.71.65:41705)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:40.733840 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.66:0
--local_ip_for_outbound_sockets=127.0.71.66
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:40553,127.0.71.85:38001,127.0.71.86:41825,127.0.71.87:42223,127.0.71.88:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:53:41.067780 20118 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:41.068081 20118 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:41.068252 20118 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:41.076741 20118 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:41.076920 20118 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.66
I20260929 01:53:41.087563 20118 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:40553,127.0.71.85:38001,127.0.71.86:41825,127.0.71.87:42223,127.0.71.88:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.71.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:41.089169 20118 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:41.090453 20118 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:41.100615 20123 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:41.100929 20126 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:41.100605 20124 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:41.101909 20118 server_base.cc:1061] running on GCE node
I20260929 01:53:41.102520 20118 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:41.103770 20118 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:41.105017 20118 hybrid_clock.cc:648] HybridClock initialized: now 1790646821104900 us; error 86 us; skew 500 ppm
I20260929 01:53:41.105366 20118 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:41 dist-test-slave-vz23 krb5kdc[19663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:41.109004 20118 init.cc:377] Logged in from keytab as kudu/127.0.71.66@KRBTEST.COM (short username kudu)
I20260929 01:53:41.111019 20118 webserver.cc:533] Webserver started at http://127.0.71.66:41557/ using document root <none> and password file <none>
I20260929 01:53:41.111815 20118 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:41.111979 20118 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:41.112370 20118 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:53:41.115093 20118 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/data/instance:
uuid: "d39478ed416a473d8959c68989748cc3"
format_stamp: "Formatted at 2026-09-29 01:53:41 on dist-test-slave-vz23"
I20260929 01:53:41.116019 20118 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/wal/instance:
uuid: "d39478ed416a473d8959c68989748cc3"
format_stamp: "Formatted at 2026-09-29 01:53:41 on dist-test-slave-vz23"
I20260929 01:53:41.122031 20118 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.005s sys 0.003s
I20260929 01:53:41.125880 20133 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:41.127369 20118 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:53:41.127508 20118 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/wal
uuid: "d39478ed416a473d8959c68989748cc3"
format_stamp: "Formatted at 2026-09-29 01:53:41 on dist-test-slave-vz23"
I20260929 01:53:41.127667 20118 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:41.145076 20118 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:41.149691 20118 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:41.150029 20118 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:41.151005 20118 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:41.152595 20118 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:41.152671 20118 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:41.152833 20118 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:41.152885 20118 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:41.187258 20118 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.66:46225
I20260929 01:53:41.187353 20245 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.66:46225 every 8 connection(s)
I20260929 01:53:41.188807 20118 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
I20260929 01:53:41.195151 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 20118
I20260929 01:53:41.195410 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/wal/instance
Sep 29 01:53:41 dist-test-slave-vz23 krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:53:41 dist-test-slave-vz23 krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:53:41 dist-test-slave-vz23 krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.67@KRBTEST.COM" created.
I20260929 01:53:41.234815 20246 heartbeater.cc:344] Connected to a master server at 127.0.71.124:43011
I20260929 01:53:41.235217 20246 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:41.236047 20246 heartbeater.cc:507] Master 127.0.71.124:43011 requested a full tablet report, sending...
I20260929 01:53:41.236941 20247 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43657
I20260929 01:53:41.237084 20247 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:41.237574 20247 heartbeater.cc:507] Master 127.0.71.126:43657 requested a full tablet report, sending...
I20260929 01:53:41.237838 19872 ts_manager.cc:194] Registered new tserver with Master: d39478ed416a473d8959c68989748cc3 (127.0.71.66:46225)
I20260929 01:53:41.238508 20248 heartbeater.cc:344] Connected to a master server at 127.0.71.125:36483
I20260929 01:53:41.238641 20248 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:41.238956 19719 ts_manager.cc:194] Registered new tserver with Master: d39478ed416a473d8959c68989748cc3 (127.0.71.66:46225)
I20260929 01:53:41.238955 19872 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.66@KRBTEST.COM'} at 127.0.71.66:49533
I20260929 01:53:41.239060 20248 heartbeater.cc:507] Master 127.0.71.125:36483 requested a full tablet report, sending...
I20260929 01:53:41.240147 19795 ts_manager.cc:194] Registered new tserver with Master: d39478ed416a473d8959c68989748cc3 (127.0.71.66:46225)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:41.316147 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.67:0
--local_ip_for_outbound_sockets=127.0.71.67
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:40553,127.0.71.85:38001,127.0.71.86:41825,127.0.71.87:42223,127.0.71.88:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:53:41.651600 20260 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:41.651937 20260 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:41.652096 20260 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:41.660343 20260 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:41.660580 20260 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.67
I20260929 01:53:41.671164 20260 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:40553,127.0.71.85:38001,127.0.71.86:41825,127.0.71.87:42223,127.0.71.88:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.71.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:41.672888 20104 heartbeater.cc:499] Master 127.0.71.124:43011 was elected leader, sending a full tablet report...
I20260929 01:53:41.672936 20260 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:41.674567 20260 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:41.683794 20266 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:41.683832 20265 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:41.684321 20268 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:41.685031 20260 server_base.cc:1061] running on GCE node
I20260929 01:53:41.685555 20260 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:41.686638 20260 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:41.687866 20260 hybrid_clock.cc:648] HybridClock initialized: now 1790646821687744 us; error 91 us; skew 500 ppm
I20260929 01:53:41.688189 20260 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:41 dist-test-slave-vz23 krb5kdc[19663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:41.691452 20260 init.cc:377] Logged in from keytab as kudu/127.0.71.67@KRBTEST.COM (short username kudu)
I20260929 01:53:41.693164 20260 webserver.cc:533] Webserver started at http://127.0.71.67:38675/ using document root <none> and password file <none>
I20260929 01:53:41.693854 20260 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:41.693990 20260 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:41.694331 20260 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:53:41.697009 20260 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/data/instance:
uuid: "88e3e35f4e6f4b46880366a101883381"
format_stamp: "Formatted at 2026-09-29 01:53:41 on dist-test-slave-vz23"
I20260929 01:53:41.697937 20260 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/wal/instance:
uuid: "88e3e35f4e6f4b46880366a101883381"
format_stamp: "Formatted at 2026-09-29 01:53:41 on dist-test-slave-vz23"
I20260929 01:53:41.703681 20260 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:53:41.707289 20275 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:41.708673 20260 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 01:53:41.708853 20260 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/wal
uuid: "88e3e35f4e6f4b46880366a101883381"
format_stamp: "Formatted at 2026-09-29 01:53:41 on dist-test-slave-vz23"
I20260929 01:53:41.709055 20260 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:41.729975 20260 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:41.733989 20260 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:41.734273 20260 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:41.735155 20260 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:41.736725 20260 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:41.736832 20260 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:41.736974 20260 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:41.737043 20260 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:41.771412 20260 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.67:42387
I20260929 01:53:41.771510 20387 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.67:42387 every 8 connection(s)
I20260929 01:53:41.772959 20260 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
I20260929 01:53:41.778261 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 20260
I20260929 01:53:41.778460 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/wal/instance
Sep 29 01:53:41 dist-test-slave-vz23 krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:53:41 dist-test-slave-vz23 krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:53:41 dist-test-slave-vz23 krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646821, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:53:41.818660 20390 heartbeater.cc:344] Connected to a master server at 127.0.71.125:36483
I20260929 01:53:41.818696 20388 heartbeater.cc:344] Connected to a master server at 127.0.71.124:43011
I20260929 01:53:41.819070 20388 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:41.819072 20390 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:41.819150 20389 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43657
I20260929 01:53:41.819316 20389 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:41.820134 20388 heartbeater.cc:507] Master 127.0.71.124:43011 requested a full tablet report, sending...
I20260929 01:53:41.820269 20390 heartbeater.cc:507] Master 127.0.71.125:36483 requested a full tablet report, sending...
I20260929 01:53:41.820286 20389 heartbeater.cc:507] Master 127.0.71.126:43657 requested a full tablet report, sending...
I20260929 01:53:41.821815 19872 ts_manager.cc:194] Registered new tserver with Master: 88e3e35f4e6f4b46880366a101883381 (127.0.71.67:42387)
I20260929 01:53:41.821954 19719 ts_manager.cc:194] Registered new tserver with Master: 88e3e35f4e6f4b46880366a101883381 (127.0.71.67:42387)
I20260929 01:53:41.822800 19872 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.67@KRBTEST.COM'} at 127.0.71.67:41695
I20260929 01:53:41.823596 19795 ts_manager.cc:194] Registered new tserver with Master: 88e3e35f4e6f4b46880366a101883381 (127.0.71.67:42387)
I20260929 01:53:41.831851 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:53:41.838395 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:53:41.838581 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 19687
I20260929 01:53:41.839396 19749 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:53:41.950546 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 19687
I20260929 01:53:41.974831 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:43657
--webserver_interface=127.0.71.126
--webserver_port=33127
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:40553,127.0.71.85:38001,127.0.71.86:41825,127.0.71.87:42223,127.0.71.88:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:42.241572 20246 heartbeater.cc:499] Master 127.0.71.124:43011 was elected leader, sending a full tablet report...
W20260929 01:53:42.243889 20247 heartbeater.cc:646] Failed to heartbeat to 127.0.71.126:43657 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.71.126:43657: connect: Connection refused (error 111)
W20260929 01:53:42.320205 20399 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:42.320557 20399 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:42.320663 20399 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:42.328979 20399 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:53:42.329074 20399 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:42.329128 20399 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:53:42.329183 20399 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:53:42.339931 20399 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:40553,127.0.71.85:38001,127.0.71.86:41825,127.0.71.87:42223,127.0.71.88:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:43657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=33127
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:42.341460 20399 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:42.342782 20399 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:42.351266 20405 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:42.351305 20406 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:42.351509 20408 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:42.352494 20399 server_base.cc:1061] running on GCE node
I20260929 01:53:42.353223 20399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:42.354774 20399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:42.356117 20399 hybrid_clock.cc:648] HybridClock initialized: now 1790646822355967 us; error 110 us; skew 500 ppm
I20260929 01:53:42.356555 20399 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20260929 01:53:42.357167 19856 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.126:43657: connect: Connection refused (error 111)
Sep 29 01:53:42 dist-test-slave-vz23 krb5kdc[19663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:42.360071 20399 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
W20260929 01:53:42.360097 19856 consensus_peers.cc:597] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff -> Peer ce7ce17b42f04948b67de74b41285e11 (127.0.71.126:43657): Couldn't send request to peer ce7ce17b42f04948b67de74b41285e11. Status: Network error: Client connection negotiation failed: client connection to 127.0.71.126:43657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 01:53:42.361830 20399 webserver.cc:533] Webserver started at http://127.0.71.126:33127/ using document root <none> and password file <none>
I20260929 01:53:42.362562 20399 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:42.362699 20399 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:42.368073 20399 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:53:42.370891 20417 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:42.372269 20399 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 01:53:42.372427 20399 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "ce7ce17b42f04948b67de74b41285e11"
format_stamp: "Formatted at 2026-09-29 01:53:38 on dist-test-slave-vz23"
I20260929 01:53:42.373205 20399 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:42.388094 20399 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:42.391810 20399 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:42.392076 20399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:42.412940 20399 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:43657
I20260929 01:53:42.412977 20468 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:43657 every 8 connection(s)
I20260929 01:53:42.414686 20399 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:53:42.416316 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 20399
I20260929 01:53:42.416728 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 19976
I20260929 01:53:42.423905 20469 sys_catalog.cc:263] Verifying existing consensus state
Sep 29 01:53:42 dist-test-slave-vz23 krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646817, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 01:53:42.427847 20469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: Bootstrap starting.
I20260929 01:53:42.436143 20099 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:53:42.449249 20469 log.cc:826] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: Log is configured to *not* fsync() on all Append() calls
I20260929 01:53:42.458509 20469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:53:42.459096 20469 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: Bootstrap complete.
I20260929 01:53:42.466269 20469 raft_consensus.cc:359] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } }
I20260929 01:53:42.467005 20469 raft_consensus.cc:740] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce7ce17b42f04948b67de74b41285e11, State: Initialized, Role: FOLLOWER
I20260929 01:53:42.467792 20469 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } }
I20260929 01:53:42.469121 20475 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:42.469491 20475 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:42.470293 20469 sys_catalog.cc:565] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:53:42.483403 20486 catalog_manager.cc:1308] Loaded cluster ID: 1d32bc027e374184a8e577eeec0bb9fd
I20260929 01:53:42.483603 20486 catalog_manager.cc:1601] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: loading cluster ID for follower catalog manager: success
I20260929 01:53:42.487710 20486 catalog_manager.cc:1623] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: acquiring CA information for follower catalog manager: success
I20260929 01:53:42.490383 20486 catalog_manager.cc:1651] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:53:42.547374 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 19976
I20260929 01:53:42.565114 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:41705
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--webserver_port=39543
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:40553,127.0.71.85:38001,127.0.71.86:41825,127.0.71.87:42223,127.0.71.88:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:42.825150 20388 heartbeater.cc:499] Master 127.0.71.124:43011 was elected leader, sending a full tablet report...
I20260929 01:53:42.835280 20434 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "88e3e35f4e6f4b46880366a101883381" instance_seqno: 1790646821763176) as {username='kudu', principal='kudu/127.0.71.67@KRBTEST.COM'} at 127.0.71.67:57429; Asking this server to re-register.
I20260929 01:53:42.836316 20389 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:42.838325 20434 ts_manager.cc:194] Registered new tserver with Master: 88e3e35f4e6f4b46880366a101883381 (127.0.71.67:42387)
I20260929 01:53:42.886909 19963 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8cf5932291314506937aeaa8e67f3dff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:42.887179 19963 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [sys.catalog]: This master's current role is: LEADER
I20260929 01:53:42.899992 20475 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cf5932291314506937aeaa8e67f3dff. Latest consensus state: current_term: 1 leader_uuid: "8cf5932291314506937aeaa8e67f3dff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:42.900264 20475 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:53:42.914958 20492 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:42.915326 20492 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:42.915494 20492 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:42.924371 20492 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:42.924721 20492 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:53:42.936717 20492 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:40553,127.0.71.85:38001,127.0.71.86:41825,127.0.71.87:42223,127.0.71.88:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:41705
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=39543
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:42.938331 20492 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:42.939682 20492 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:42.949540 20502 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:42.949540 20503 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:42.950815 20505 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:42.952680 20492 server_base.cc:1061] running on GCE node
I20260929 01:53:42.953290 20492 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:42.954507 20492 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:42.955732 20492 hybrid_clock.cc:648] HybridClock initialized: now 1790646822955603 us; error 97 us; skew 500 ppm
I20260929 01:53:42.956053 20492 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:42 dist-test-slave-vz23 krb5kdc[19663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:42.959467 20492 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:53:42.961208 20492 webserver.cc:533] Webserver started at http://127.0.71.65:39543/ using document root <none> and password file <none>
I20260929 01:53:42.961897 20492 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:42.962003 20492 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:42.967463 20492 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 01:53:42.970290 20512 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:42.971750 20492 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:53:42.971954 20492 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "ca2810190b0f4b078d7d671be7879084"
format_stamp: "Formatted at 2026-09-29 01:53:40 on dist-test-slave-vz23"
I20260929 01:53:42.972764 20492 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:42.991555 20492 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:42.995541 20492 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:42.995831 20492 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:42.996824 20492 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:42.998461 20492 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:42.998548 20492 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:42.998699 20492 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:42.998804 20492 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:43.042948 20492 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:41705
I20260929 01:53:43.043042 20624 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:41705 every 8 connection(s)
I20260929 01:53:43.044626 20492 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:53:43.045462 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 20492
Sep 29 01:53:43 dist-test-slave-vz23 krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:53:43 dist-test-slave-vz23 krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:53:43 dist-test-slave-vz23 krb5kdc[19663](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646822, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:53:43.083884 20626 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43657
I20260929 01:53:43.084291 20626 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:43.085500 20626 heartbeater.cc:507] Master 127.0.71.126:43657 requested a full tablet report, sending...
I20260929 01:53:43.087653 20434 ts_manager.cc:194] Registered new tserver with Master: ca2810190b0f4b078d7d671be7879084 (127.0.71.65:41705)
I20260929 01:53:43.090716 20625 heartbeater.cc:344] Connected to a master server at 127.0.71.124:43011
I20260929 01:53:43.090955 20627 heartbeater.cc:344] Connected to a master server at 127.0.71.125:36483
I20260929 01:53:43.091099 20627 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:43.091586 20627 heartbeater.cc:507] Master 127.0.71.125:36483 requested a full tablet report, sending...
I20260929 01:53:43.090969 20625 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:43.092387 20625 heartbeater.cc:507] Master 127.0.71.124:43011 requested a full tablet report, sending...
I20260929 01:53:43.092986 19795 ts_manager.cc:194] Re-registered known tserver with Master: ca2810190b0f4b078d7d671be7879084 (127.0.71.65:41705)
I20260929 01:53:43.093806 19872 ts_manager.cc:194] Re-registered known tserver with Master: ca2810190b0f4b078d7d671be7879084 (127.0.71.65:41705)
I20260929 01:53:43.094964 19872 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:42309
I20260929 01:53:43.102061 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 20492
W20260929 01:53:43.103331 20637 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:43.098505 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:55378 (local address 127.0.71.65:41705)
0929 01:53:43.100388 (+ 1883us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:43.100395 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:43.100421 (+ 26us) server_negotiation.cc:408] Connection header received
0929 01:53:43.100531 (+ 110us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:53:43.100538 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:53:43.100735 (+ 197us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:53:43.100933 (+ 198us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:53:43.101916 (+ 983us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55378: BlockingRecv error: recv got EOF from 127.0.0.1:55378 (error 108)
Metrics: {"server-negotiator.queue_time_us":1768,"thread_start_us":1595,"threads_started":1}
W20260929 01:53:43.104295 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 20492: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:41705: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:53:43.104480 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 20492
I20260929 01:53:43.122464 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:41705
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--webserver_port=39543
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:40553,127.0.71.85:38001,127.0.71.86:41825,127.0.71.87:42223,127.0.71.88:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:43.251289 20247 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43657
I20260929 01:53:43.252308 20434 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d39478ed416a473d8959c68989748cc3" instance_seqno: 1790646821179189) as {username='kudu', principal='kudu/127.0.71.66@KRBTEST.COM'} at 127.0.71.66:51853; Asking this server to re-register.
I20260929 01:53:43.252907 20247 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:43.254096 20434 ts_manager.cc:194] Registered new tserver with Master: d39478ed416a473d8959c68989748cc3 (127.0.71.66:46225)
I20260929 01:53:43.428875 20495 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "8cf5932291314506937aeaa8e67f3dff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:43.429155 20495 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [sys.catalog]: This master's current role is: LEADER
W20260929 01:53:43.460556 20639 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:43.460934 20639 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:43.461110 20639 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:53:43.461215 20639 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:43.469874 20639 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:43.470182 20639 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:53:43.483444 20639 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:40553,127.0.71.85:38001,127.0.71.86:41825,127.0.71.87:42223,127.0.71.88:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:41705
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=39543
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:43.485278 20639 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:43.486672 20639 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:43.497058 20648 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:43.497128 20649 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:43.497457 20651 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:43.498848 20639 server_base.cc:1061] running on GCE node
I20260929 01:53:43.499470 20639 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:43.500820 20639 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:43.502084 20639 hybrid_clock.cc:648] HybridClock initialized: now 1790646823501903 us; error 129 us; skew 500 ppm
I20260929 01:53:43.502417 20639 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:43 dist-test-slave-vz23 krb5kdc[19663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:43.506507 20639 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:53:43.512768 20639 webserver.cc:533] Webserver started at http://127.0.71.65:39543/ using document root <none> and password file <none>
I20260929 01:53:43.513746 20639 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:43.513845 20639 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:43.520771 20639 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:53:43.523418 20658 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:43.525009 20639 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:53:43.525192 20639 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "ca2810190b0f4b078d7d671be7879084"
format_stamp: "Formatted at 2026-09-29 01:53:40 on dist-test-slave-vz23"
I20260929 01:53:43.525907 20639 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:43.554450 20639 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:43.558755 20639 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:43.559077 20639 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:43.560086 20639 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:43.561709 20639 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:43.561830 20639 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:43.561990 20639 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:43.562062 20639 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:53:43.562445 20639 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:53:43.562538 20639 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:53:43.562623 20639 fault_injection.cc:59] injecting fault for kudu (pid 20639): FLAGS_fault_before_start (process will exit)
I20260929 01:53:43.584815 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:41705
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--webserver_port=39543
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:40553,127.0.71.85:38001,127.0.71.86:41825,127.0.71.87:42223,127.0.71.88:44719
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:53:43.929966 20666 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:43.930289 20666 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:43.930501 20666 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:43.939730 20666 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:43.940003 20666 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:53:43.957021 20666 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:40553,127.0.71.85:38001,127.0.71.86:41825,127.0.71.87:42223,127.0.71.88:44719
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:41705
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=39543
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:43.958693 20666 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:43.960121 20666 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:43.970186 20671 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:43.970209 20672 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:43.970871 20674 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:43.971446 20666 server_base.cc:1061] running on GCE node
I20260929 01:53:43.972043 20666 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:43.973234 20666 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:43.974466 20666 hybrid_clock.cc:648] HybridClock initialized: now 1790646823974327 us; error 112 us; skew 500 ppm
I20260929 01:53:43.974814 20666 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:43 dist-test-slave-vz23 krb5kdc[19663](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646823, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:43.978353 20666 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:53:43.980000 20666 webserver.cc:533] Webserver started at http://127.0.71.65:39543/ using document root <none> and password file <none>
I20260929 01:53:43.980809 20666 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:43.980944 20666 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:43.986244 20666 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.001s
I20260929 01:53:43.988880 20681 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:43.990182 20666 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:53:43.990352 20666 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "ca2810190b0f4b078d7d671be7879084"
format_stamp: "Formatted at 2026-09-29 01:53:40 on dist-test-slave-vz23"
I20260929 01:53:43.991051 20666 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:44.009020 20666 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:44.012845 20666 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:44.013116 20666 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:44.014173 20666 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:44.015694 20666 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:44.015830 20666 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:44.015960 20666 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:44.016049 20666 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:44.050792 20666 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:41705
I20260929 01:53:44.050827 20793 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:41705 every 8 connection(s)
I20260929 01:53:44.052343 20666 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:53:44.057713 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 20666
I20260929 01:53:44.058079 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 20666
W20260929 01:53:44.081610 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 20666: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:41705: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:53:44.081836 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 20666
I20260929 01:53:44.098680 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 20118
W20260929 01:53:44.099457 20802 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:44.057757 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:55625 (local address 127.0.71.126:43657)
0929 01:53:44.083215 (+ 25458us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:44.083221 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:44.083243 (+ 22us) server_negotiation.cc:408] Connection header received
0929 01:53:44.083320 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:53:44.083327 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:53:44.083412 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:53:44.083519 (+ 107us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:53:44.098476 (+ 14957us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:55625: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":25331,"thread_start_us":25357,"threads_started":1}
W20260929 01:53:44.099897 20798 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:44.055759 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:42413 (local address 127.0.71.125:36483)
0929 01:53:44.078654 (+ 22895us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:44.078664 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:44.098471 (+ 19807us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:42413: BlockingRecv error: recv got EOF from 127.0.71.65:42413 (error 108)
Metrics: {"server-negotiator.queue_time_us":22712,"thread_start_us":22617,"threads_started":1}
W20260929 01:53:44.100044 20799 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:44.055878 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:47589 (local address 127.0.71.124:43011)
0929 01:53:44.079050 (+ 23172us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:44.079057 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:44.079079 (+ 22us) server_negotiation.cc:408] Connection header received
0929 01:53:44.079157 (+ 78us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:53:44.079165 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:53:44.079273 (+ 108us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:53:44.079426 (+ 153us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:53:44.099120 (+ 19694us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:47589: BlockingRecv error: recv got EOF from 127.0.71.65:47589 (error 108)
Metrics: {"server-negotiator.queue_time_us":23048,"thread_start_us":22954,"threads_started":1}
W20260929 01:53:44.101830 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 20118: Not authorized: Client connection negotiation failed: client connection to 127.0.71.66:46225: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:53:44.101995 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 20118
I20260929 01:53:44.118984 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 20260
W20260929 01:53:44.121733 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 20260: Not authorized: Client connection negotiation failed: client connection to 127.0.71.67:42387: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:53:44.121914 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 20260
I20260929 01:53:44.138345 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 20399
W20260929 01:53:44.140539 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 20399: Not authorized: Client connection negotiation failed: client connection to 127.0.71.126:43657: server requires authentication, but client does not have Kerberos credentials available
W20260929 01:53:44.140661 20802 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:44.139161 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:53558 (local address 127.0.71.126:43657)
0929 01:53:44.139372 (+ 211us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:44.139378 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:44.139400 (+ 22us) server_negotiation.cc:408] Connection header received
0929 01:53:44.139588 (+ 188us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:53:44.139594 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:53:44.139664 (+ 70us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:53:44.139766 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:53:44.140477 (+ 711us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:53558: BlockingRecv error: recv got EOF from 127.0.0.1:53558 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 01:53:44.140697 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 20399
I20260929 01:53:44.161342 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 19763
I20260929 01:53:44.162514 19825 generic_service.cc:196] Checking for leaks (request via RPC)
W20260929 01:53:44.241019 19856 consensus_peers.cc:597] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff -> Peer ce7ce17b42f04948b67de74b41285e11 (127.0.71.126:43657): Couldn't send request to peer ce7ce17b42f04948b67de74b41285e11. Status: Network error: Client connection negotiation failed: client connection to 127.0.71.126:43657: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 01:53:44.273962 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 19763
I20260929 01:53:44.295245 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 19840
I20260929 01:53:44.296526 19902 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:53:44.408041 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 19840
2026-09-29T01:53:44Z chronyd exiting
2026-09-29T01:53:44Z chronyd exiting
2026-09-29T01:53:44Z chronyd exiting
2026-09-29T01:53:44Z chronyd exiting
2026-09-29T01:53:44Z chronyd exiting
Sep 29 01:53:44 dist-test-slave-vz23 krb5kdc[20813](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:53:44 dist-test-slave-vz23 krb5kdc[20813](info): set up 2 sockets
Sep 29 01:53:44 dist-test-slave-vz23 krb5kdc[20813](info): commencing operation
krb5kdc: starting...
W20260929 01:53:46.543196 285 mini_kdc.cc:121] Time spent starting KDC: real 2.058s user 0.001s sys 0.016s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-admin@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-user@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "joe-interloper@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 01:53:46 dist-test-slave-vz23 krb5kdc[20813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:53:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:53:46Z Disabled control of system clock
2026-09-29T01:53:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:53:46Z Disabled control of system clock
2026-09-29T01:53:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:53:46Z Disabled control of system clock
2026-09-29T01:53:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:53:46Z Disabled control of system clock
2026-09-29T01:53:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:53:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.126@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.126@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:46.863633 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:43657
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:38365,127.0.71.85:32921,127.0.71.86:40115,127.0.71.87:44833,127.0.71.88:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:53:47.207518 20837 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:47.207875 20837 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:47.208035 20837 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:47.216316 20837 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:53:47.216411 20837 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:47.216583 20837 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:53:47.216673 20837 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:53:47.228106 20837 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38365,127.0.71.85:32921,127.0.71.86:40115,127.0.71.87:44833,127.0.71.88:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:43657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:47.229795 20837 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:47.231215 20837 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:47.240098 20843 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:47.240041 20842 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:47.241209 20845 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:47.241461 20837 server_base.cc:1061] running on GCE node
I20260929 01:53:47.242266 20837 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:47.243835 20837 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:47.245061 20837 hybrid_clock.cc:648] HybridClock initialized: now 1790646827244900 us; error 131 us; skew 500 ppm
I20260929 01:53:47.245409 20837 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:47 dist-test-slave-vz23 krb5kdc[20813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:47.249068 20837 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:53:47.250686 20837 webserver.cc:533] Webserver started at http://127.0.71.126:44749/ using document root <none> and password file <none>
I20260929 01:53:47.251408 20837 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:47.251546 20837 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:47.256990 20837 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 01:53:47.259724 20852 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:47.261130 20837 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:53:47.261330 20837 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "ce7ce17b42f04948b67de74b41285e11"
format_stamp: "Formatted at 2026-09-29 01:53:38 on dist-test-slave-vz23"
I20260929 01:53:47.262079 20837 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:47.278546 20837 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:47.282303 20837 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:47.282595 20837 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:47.303211 20837 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:43657
I20260929 01:53:47.303232 20903 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:43657 every 8 connection(s)
I20260929 01:53:47.304905 20837 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:53:47.311923 20904 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:53:47.315059 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 20837
I20260929 01:53:47.315303 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:53:47.315596 20904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: Bootstrap starting.
I20260929 01:53:47.335069 20904 log.cc:826] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: Log is configured to *not* fsync() on all Append() calls
WARNING: no policy specified for kudu/127.0.71.125@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.125@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:53:47.345840 20904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:53:47.346397 20904 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: Bootstrap complete.
I20260929 01:53:47.353642 20904 raft_consensus.cc:359] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } }
I20260929 01:53:47.354380 20904 raft_consensus.cc:740] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce7ce17b42f04948b67de74b41285e11, State: Initialized, Role: FOLLOWER
I20260929 01:53:47.355247 20904 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } }
I20260929 01:53:47.356179 20909 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:47.356575 20909 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:47.357230 20904 sys_catalog.cc:565] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [sys.catalog]: configured and running, proceeding with master startup.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.125 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.125 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:47.368073 20920 catalog_manager.cc:1308] Loaded cluster ID: 1d32bc027e374184a8e577eeec0bb9fd
I20260929 01:53:47.368170 20920 catalog_manager.cc:1601] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: loading cluster ID for follower catalog manager: success
I20260929 01:53:47.373006 20920 catalog_manager.cc:1623] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: acquiring CA information for follower catalog manager: success
I20260929 01:53:47.375818 20920 catalog_manager.cc:1651] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for HTTP/127.0.71.125@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.125@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:53:47.392756 20837 master_runner.cc:187] Error getting master registration for 127.0.71.125:36483: Network error: Client connection negotiation failed: client connection to 127.0.71.125:36483: connect: Connection refused (error 111),
I20260929 01:53:47.398589 20837 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.125 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.125 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:47.420321 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:36483
--webserver_interface=127.0.71.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:38365,127.0.71.85:32921,127.0.71.86:40115,127.0.71.87:44833,127.0.71.88:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:53:47.763353 20926 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:47.763728 20926 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:47.763909 20926 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:47.773523 20926 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:53:47.773635 20926 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:47.773766 20926 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:53:47.773864 20926 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:53:47.784817 20926 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38365,127.0.71.85:32921,127.0.71.86:40115,127.0.71.87:44833,127.0.71.88:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:36483
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:47.786597 20926 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:47.787919 20926 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:47.796373 20932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:47.796368 20931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:47.797055 20934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:47.797382 20926 server_base.cc:1061] running on GCE node
I20260929 01:53:47.798074 20926 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:47.799957 20926 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:47.801188 20926 hybrid_clock.cc:648] HybridClock initialized: now 1790646827801038 us; error 117 us; skew 500 ppm
I20260929 01:53:47.801568 20926 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:47 dist-test-slave-vz23 krb5kdc[20813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646827, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:47.804940 20926 init.cc:377] Logged in from keytab as kudu/127.0.71.125@KRBTEST.COM (short username kudu)
I20260929 01:53:47.806416 20926 webserver.cc:533] Webserver started at http://127.0.71.125:42397/ using document root <none> and password file <none>
I20260929 01:53:47.807104 20926 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:47.807264 20926 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:47.812737 20926 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:53:47.815652 20941 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:47.817031 20926 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:53:47.817291 20926 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "444f97234e6d4c20be102cb7c0628408"
format_stamp: "Formatted at 2026-09-29 01:53:38 on dist-test-slave-vz23"
I20260929 01:53:47.818025 20926 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:47.838909 20926 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:47.842880 20926 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:47.843194 20926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:47.864892 20926 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:36483
I20260929 01:53:47.864899 20992 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:36483 every 8 connection(s)
I20260929 01:53:47.866524 20926 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:53:47.872294 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 20926
I20260929 01:53:47.872531 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:53:47.873775 20993 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:53:47.877134 20993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408: Bootstrap starting.
WARNING: no policy specified for kudu/127.0.71.124@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.124@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:53:47.898321 20993 log.cc:826] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408: Log is configured to *not* fsync() on all Append() calls
I20260929 01:53:47.908672 20993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:53:47.909353 20993 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408: Bootstrap complete.
I20260929 01:53:47.919901 20993 raft_consensus.cc:359] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } }
I20260929 01:53:47.920732 20993 raft_consensus.cc:740] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 444f97234e6d4c20be102cb7c0628408, State: Initialized, Role: FOLLOWER
I20260929 01:53:47.921555 20993 consensus_queue.cc:260] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } }
I20260929 01:53:47.922353 20998 sys_catalog.cc:455] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:47.922765 20998 sys_catalog.cc:458] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:47.923470 20993 sys_catalog.cc:565] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [sys.catalog]: configured and running, proceeding with master startup.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.124 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.124 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:47.933890 21009 catalog_manager.cc:1308] Loaded cluster ID: 1d32bc027e374184a8e577eeec0bb9fd
I20260929 01:53:47.934065 21009 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408: loading cluster ID for follower catalog manager: success
I20260929 01:53:47.938580 21009 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408: acquiring CA information for follower catalog manager: success
I20260929 01:53:47.941236 21009 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for HTTP/127.0.71.124@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.124@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:53:47.965452 20926 master_runner.cc:187] Error getting master registration for 127.0.71.124:43011: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43011: connect: Connection refused (error 111),
I20260929 01:53:47.972653 20926 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.124 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.124 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:47.991544 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:43011
--webserver_interface=127.0.71.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:38365,127.0.71.85:32921,127.0.71.86:40115,127.0.71.87:44833,127.0.71.88:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:53:48.348201 21016 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:48.348608 21016 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:48.348783 21016 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:48.357337 21016 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:53:48.357442 21016 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:48.357581 21016 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:53:48.357669 21016 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:53:48.369055 21016 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38365,127.0.71.85:32921,127.0.71.86:40115,127.0.71.87:44833,127.0.71.88:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:43011
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:48.370637 21016 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:48.371943 21016 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:48.380486 21024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:48.380353 21021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:48.380348 21022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:48.381350 21016 server_base.cc:1061] running on GCE node
I20260929 01:53:48.382079 21016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:48.383499 21016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:48.384691 21016 hybrid_clock.cc:648] HybridClock initialized: now 1790646828384587 us; error 77 us; skew 500 ppm
I20260929 01:53:48.385102 21016 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:48 dist-test-slave-vz23 krb5kdc[20813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646828, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:48.388407 21016 init.cc:377] Logged in from keytab as kudu/127.0.71.124@KRBTEST.COM (short username kudu)
I20260929 01:53:48.390048 21016 webserver.cc:533] Webserver started at http://127.0.71.124:34993/ using document root <none> and password file <none>
I20260929 01:53:48.390776 21016 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:48.390933 21016 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:48.396270 21016 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:53:48.399430 21031 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:48.400868 21016 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:53:48.401051 21016 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "8cf5932291314506937aeaa8e67f3dff"
format_stamp: "Formatted at 2026-09-29 01:53:39 on dist-test-slave-vz23"
I20260929 01:53:48.401774 21016 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:48.416716 21016 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:48.420205 21016 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:48.420548 21016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:48.440909 21016 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:43011
I20260929 01:53:48.440929 21082 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:43011 every 8 connection(s)
I20260929 01:53:48.442420 21016 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:53:48.446422 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 21016
I20260929 01:53:48.446623 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:53:48.450651 21083 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:53:48.455204 21083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff: Bootstrap starting.
Sep 29 01:53:48 dist-test-slave-vz23 krb5kdc[20813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:53:48 dist-test-slave-vz23 krb5kdc[20813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:53:48.474282 21083 log.cc:826] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff: Log is configured to *not* fsync() on all Append() calls
I20260929 01:53:48.484614 21083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:53:48.485142 21083 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff: Bootstrap complete.
Sep 29 01:53:48 dist-test-slave-vz23 krb5kdc[20813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:53:48.492203 21083 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } }
I20260929 01:53:48.493160 21083 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8cf5932291314506937aeaa8e67f3dff, State: Initialized, Role: FOLLOWER
I20260929 01:53:48.493996 21083 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } }
I20260929 01:53:48.495312 21089 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:48.495730 21089 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:48.496639 21083 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:53:48.507809 21100 catalog_manager.cc:1308] Loaded cluster ID: 1d32bc027e374184a8e577eeec0bb9fd
I20260929 01:53:48.507918 21100 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff: loading cluster ID for follower catalog manager: success
I20260929 01:53:48.512738 21100 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff: acquiring CA information for follower catalog manager: success
I20260929 01:53:48.515362 21100 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:53:48.544235 21016 master_runner.cc:264] No leader master found from master 8cf5932291314506937aeaa8e67f3dff
I20260929 01:53:48.544366 21016 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:53:48.943949 21103 raft_consensus.cc:493] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:53:48.944244 21103 raft_consensus.cc:515] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } }
I20260929 01:53:48.946432 21103 leader_election.cc:290] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 444f97234e6d4c20be102cb7c0628408 (127.0.71.125:36483), 8cf5932291314506937aeaa8e67f3dff (127.0.71.124:43011)
I20260929 01:53:48.955322 20968 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce7ce17b42f04948b67de74b41285e11" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "444f97234e6d4c20be102cb7c0628408" is_pre_election: true
I20260929 01:53:48.955474 21058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce7ce17b42f04948b67de74b41285e11" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8cf5932291314506937aeaa8e67f3dff" is_pre_election: true
I20260929 01:53:48.955885 20968 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce7ce17b42f04948b67de74b41285e11 in term 1.
I20260929 01:53:48.956001 21058 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ce7ce17b42f04948b67de74b41285e11 in term 1.
I20260929 01:53:48.956568 20854 leader_election.cc:304] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 444f97234e6d4c20be102cb7c0628408, ce7ce17b42f04948b67de74b41285e11; no voters:
I20260929 01:53:48.956975 21103 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 01:53:48.957154 21103 raft_consensus.cc:493] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:53:48.957273 21103 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:53:48.961143 21103 raft_consensus.cc:515] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } }
I20260929 01:53:48.961817 21103 leader_election.cc:290] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [CANDIDATE]: Term 2 election: Requested vote from peers 444f97234e6d4c20be102cb7c0628408 (127.0.71.125:36483), 8cf5932291314506937aeaa8e67f3dff (127.0.71.124:43011)
I20260929 01:53:48.962249 20968 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce7ce17b42f04948b67de74b41285e11" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "444f97234e6d4c20be102cb7c0628408"
I20260929 01:53:48.962383 21058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ce7ce17b42f04948b67de74b41285e11" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "8cf5932291314506937aeaa8e67f3dff"
I20260929 01:53:48.962456 20968 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:53:48.962596 21058 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:53:48.966741 21058 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce7ce17b42f04948b67de74b41285e11 in term 2.
I20260929 01:53:48.966796 20968 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ce7ce17b42f04948b67de74b41285e11 in term 2.
I20260929 01:53:48.967197 20856 leader_election.cc:304] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8cf5932291314506937aeaa8e67f3dff, ce7ce17b42f04948b67de74b41285e11; no voters:
I20260929 01:53:48.967512 21103 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:53:48.968118 21103 raft_consensus.cc:697] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [term 2 LEADER]: Becoming Leader. State: Replica: ce7ce17b42f04948b67de74b41285e11, State: Running, Role: LEADER
I20260929 01:53:48.968680 21103 consensus_queue.cc:237] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } }
I20260929 01:53:48.971369 21106 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce7ce17b42f04948b67de74b41285e11. Latest consensus state: current_term: 2 leader_uuid: "ce7ce17b42f04948b67de74b41285e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:48.971699 21106 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [sys.catalog]: This master's current role is: LEADER
I20260929 01:53:48.972390 21108 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:53:48.974797 21108 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:53:48.975646 21108 catalog_manager.cc:1308] Loaded cluster ID: 1d32bc027e374184a8e577eeec0bb9fd
I20260929 01:53:48.975720 21108 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:53:48.976965 21108 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:53:48.977775 21108 catalog_manager.cc:6230] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: Loaded TSK: 0
I20260929 01:53:48.978669 21108 catalog_manager.cc:1563] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.65@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.65@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:49.095850 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:38365,127.0.71.85:32921,127.0.71.86:40115,127.0.71.87:44833,127.0.71.88:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:49.389079 20968 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [term 2 FOLLOWER]: Refusing update from remote peer ce7ce17b42f04948b67de74b41285e11: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 01:53:49.390187 21106 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:53:49.393748 21114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce7ce17b42f04948b67de74b41285e11. Latest consensus state: current_term: 2 leader_uuid: "ce7ce17b42f04948b67de74b41285e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:49.394115 21114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:49.396311 21103 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ce7ce17b42f04948b67de74b41285e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:49.396675 21103 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [sys.catalog]: This master's current role is: LEADER
I20260929 01:53:49.396979 21114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ce7ce17b42f04948b67de74b41285e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:49.397189 21114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:49.397368 21058 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [term 2 FOLLOWER]: Refusing update from remote peer ce7ce17b42f04948b67de74b41285e11: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 01:53:49.398196 21106 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:53:49.404858 21121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ce7ce17b42f04948b67de74b41285e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:49.405169 21121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:49.405730 21118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [sys.catalog]: SysCatalogTable state changed. Reason: New leader ce7ce17b42f04948b67de74b41285e11. Latest consensus state: current_term: 2 leader_uuid: "ce7ce17b42f04948b67de74b41285e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:49.405984 21118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:53:49.431851 21113 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:49.432196 21113 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:49.432386 21113 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:49.440838 21113 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:49.441067 21113 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:53:49.451884 21113 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38365,127.0.71.85:32921,127.0.71.86:40115,127.0.71.87:44833,127.0.71.88:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:49.453639 21113 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:49.455150 21113 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:49.464903 21127 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:49.464994 21126 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:49.466455 21129 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:49.467211 21113 server_base.cc:1061] running on GCE node
I20260929 01:53:49.467800 21113 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:49.469010 21113 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:49.470255 21113 hybrid_clock.cc:648] HybridClock initialized: now 1790646829470073 us; error 154 us; skew 500 ppm
I20260929 01:53:49.470679 21113 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:49 dist-test-slave-vz23 krb5kdc[20813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:49.474432 21113 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:53:49.476089 21113 webserver.cc:533] Webserver started at http://127.0.71.65:37987/ using document root <none> and password file <none>
I20260929 01:53:49.476876 21113 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:49.477015 21113 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:49.482472 21113 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:53:49.485261 21136 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:49.486598 21113 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:53:49.486781 21113 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "ca2810190b0f4b078d7d671be7879084"
format_stamp: "Formatted at 2026-09-29 01:53:40 on dist-test-slave-vz23"
I20260929 01:53:49.487488 21113 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:49.516371 21113 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:49.520805 21113 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:49.521044 21113 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:49.521991 21113 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:49.523800 21113 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:49.523895 21113 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:49.523978 21113 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:49.524041 21113 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:49.561046 21113 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:36053
I20260929 01:53:49.561210 21248 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:36053 every 8 connection(s)
I20260929 01:53:49.562695 21113 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:53:49.568733 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 21113
I20260929 01:53:49.568953 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
Sep 29 01:53:49 dist-test-slave-vz23 krb5kdc[20813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:53:49 dist-test-slave-vz23 krb5kdc[20813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:53:49 dist-test-slave-vz23 krb5kdc[20813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646829, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:53:49.595204 21251 heartbeater.cc:344] Connected to a master server at 127.0.71.125:36483
I20260929 01:53:49.595398 21250 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43657
I20260929 01:53:49.595726 21251 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:49.595933 21249 heartbeater.cc:344] Connected to a master server at 127.0.71.124:43011
I20260929 01:53:49.596087 21249 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:49.595724 21250 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:49.596827 21249 heartbeater.cc:507] Master 127.0.71.124:43011 requested a full tablet report, sending...
I20260929 01:53:49.596895 21251 heartbeater.cc:507] Master 127.0.71.125:36483 requested a full tablet report, sending...
I20260929 01:53:49.597404 21250 heartbeater.cc:507] Master 127.0.71.126:43657 requested a full tablet report, sending...
I20260929 01:53:49.599568 20958 ts_manager.cc:194] Registered new tserver with Master: ca2810190b0f4b078d7d671be7879084 (127.0.71.65:36053)
I20260929 01:53:49.599593 21047 ts_manager.cc:194] Registered new tserver with Master: ca2810190b0f4b078d7d671be7879084 (127.0.71.65:36053)
I20260929 01:53:49.599555 20869 ts_manager.cc:194] Registered new tserver with Master: ca2810190b0f4b078d7d671be7879084 (127.0.71.65:36053)
I20260929 01:53:49.601725 20869 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:38723
WARNING: no policy specified for kudu/127.0.71.66@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.66@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.66 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.66 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.66@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.66@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.66 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.66 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:49.679158 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.66:0
--local_ip_for_outbound_sockets=127.0.71.66
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:38365,127.0.71.85:32921,127.0.71.86:40115,127.0.71.87:44833,127.0.71.88:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:49.931257 21106 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ce7ce17b42f04948b67de74b41285e11" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:49.931625 21106 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [sys.catalog]: This master's current role is: LEADER
W20260929 01:53:50.019315 21263 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:50.019693 21263 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:50.019822 21263 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:50.028339 21263 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:50.028586 21263 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.66
I20260929 01:53:50.039273 21263 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38365,127.0.71.85:32921,127.0.71.86:40115,127.0.71.87:44833,127.0.71.88:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.71.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:50.040956 21263 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:50.042301 21263 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:50.052057 21272 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:50.052495 21271 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:50.052683 21274 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:50.053439 21263 server_base.cc:1061] running on GCE node
I20260929 01:53:50.054111 21263 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:50.055362 21263 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:50.056697 21263 hybrid_clock.cc:648] HybridClock initialized: now 1790646830056566 us; error 98 us; skew 500 ppm
I20260929 01:53:50.057096 21263 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:50 dist-test-slave-vz23 krb5kdc[20813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:50.061161 21263 init.cc:377] Logged in from keytab as kudu/127.0.71.66@KRBTEST.COM (short username kudu)
I20260929 01:53:50.062837 21263 webserver.cc:533] Webserver started at http://127.0.71.66:33163/ using document root <none> and password file <none>
I20260929 01:53:50.063730 21263 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:50.063911 21263 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:50.070277 21263 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 01:53:50.073567 21281 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:50.075064 21263 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:53:50.075322 21263 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/wal
uuid: "d39478ed416a473d8959c68989748cc3"
format_stamp: "Formatted at 2026-09-29 01:53:41 on dist-test-slave-vz23"
I20260929 01:53:50.076092 21263 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:50.103801 21263 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:50.112816 21263 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:50.113116 21263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:50.114164 21263 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:50.115783 21263 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:50.115911 21263 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:50.116043 21263 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:50.116132 21263 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:50.155774 21263 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.66:46353
I20260929 01:53:50.155825 21393 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.66:46353 every 8 connection(s)
I20260929 01:53:50.157503 21263 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
I20260929 01:53:50.162690 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 21263
I20260929 01:53:50.162952 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-1/wal/instance
WARNING: no policy specified for kudu/127.0.71.67@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.67@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Sep 29 01:53:50 dist-test-slave-vz23 krb5kdc[20813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:53:50 dist-test-slave-vz23 krb5kdc[20813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:53:50 dist-test-slave-vz23 krb5kdc[20813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:53:50.238766 21396 heartbeater.cc:344] Connected to a master server at 127.0.71.125:36483
I20260929 01:53:50.239243 21396 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:50.239883 21394 heartbeater.cc:344] Connected to a master server at 127.0.71.124:43011
I20260929 01:53:50.240015 21394 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:50.240517 21396 heartbeater.cc:507] Master 127.0.71.125:36483 requested a full tablet report, sending...
I20260929 01:53:50.240599 21394 heartbeater.cc:507] Master 127.0.71.124:43011 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.67 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.67 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:50.242641 20958 ts_manager.cc:194] Registered new tserver with Master: d39478ed416a473d8959c68989748cc3 (127.0.71.66:46353)
I20260929 01:53:50.242640 21047 ts_manager.cc:194] Registered new tserver with Master: d39478ed416a473d8959c68989748cc3 (127.0.71.66:46353)
I20260929 01:53:50.247246 21395 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43657
I20260929 01:53:50.247399 21395 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:50.247831 21395 heartbeater.cc:507] Master 127.0.71.126:43657 requested a full tablet report, sending...
I20260929 01:53:50.249024 20869 ts_manager.cc:194] Registered new tserver with Master: d39478ed416a473d8959c68989748cc3 (127.0.71.66:46353)
I20260929 01:53:50.250140 20869 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.66@KRBTEST.COM'} at 127.0.71.66:48249
WARNING: no policy specified for HTTP/127.0.71.67@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.67@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.67 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.67 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:53:50.289593 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.67:0
--local_ip_for_outbound_sockets=127.0.71.67
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:38365,127.0.71.85:32921,127.0.71.86:40115,127.0.71.87:44833,127.0.71.88:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:50.604542 21250 heartbeater.cc:499] Master 127.0.71.126:43657 was elected leader, sending a full tablet report...
W20260929 01:53:50.632031 21408 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:50.632339 21408 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:50.632432 21408 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:50.640705 21408 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:50.640887 21408 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.67
I20260929 01:53:50.651746 21408 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38365,127.0.71.85:32921,127.0.71.86:40115,127.0.71.87:44833,127.0.71.88:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.71.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:50.653462 21408 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:50.654693 21408 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:50.664659 21414 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:50.664749 21413 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:50.665474 21416 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:50.667034 21408 server_base.cc:1061] running on GCE node
I20260929 01:53:50.667692 21408 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:50.668829 21408 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:50.670029 21408 hybrid_clock.cc:648] HybridClock initialized: now 1790646830669904 us; error 95 us; skew 500 ppm
I20260929 01:53:50.670380 21408 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:50 dist-test-slave-vz23 krb5kdc[20813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:50.674018 21408 init.cc:377] Logged in from keytab as kudu/127.0.71.67@KRBTEST.COM (short username kudu)
I20260929 01:53:50.675594 21408 webserver.cc:533] Webserver started at http://127.0.71.67:42933/ using document root <none> and password file <none>
I20260929 01:53:50.676416 21408 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:50.676731 21408 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:50.682778 21408 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.003s
I20260929 01:53:50.685611 21423 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:50.687003 21408 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:53:50.687139 21408 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/wal
uuid: "88e3e35f4e6f4b46880366a101883381"
format_stamp: "Formatted at 2026-09-29 01:53:41 on dist-test-slave-vz23"
I20260929 01:53:50.687799 21408 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:50.704808 21408 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:50.708717 21408 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:50.708989 21408 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:50.709939 21408 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:50.711390 21408 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:50.711474 21408 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:50.711553 21408 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:50.711607 21408 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:50.748412 21408 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.67:41347
I20260929 01:53:50.748534 21535 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.67:41347 every 8 connection(s)
I20260929 01:53:50.750044 21408 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
I20260929 01:53:50.752153 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 21408
I20260929 01:53:50.752386 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-2/wal/instance
Sep 29 01:53:50 dist-test-slave-vz23 krb5kdc[20813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:53:50 dist-test-slave-vz23 krb5kdc[20813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:53:50 dist-test-slave-vz23 krb5kdc[20813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646830, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:53:50.786072 21538 heartbeater.cc:344] Connected to a master server at 127.0.71.125:36483
I20260929 01:53:50.786237 21537 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43657
I20260929 01:53:50.786458 21538 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:50.786463 21537 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:50.787215 21536 heartbeater.cc:344] Connected to a master server at 127.0.71.124:43011
I20260929 01:53:50.787364 21536 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:50.787559 21538 heartbeater.cc:507] Master 127.0.71.125:36483 requested a full tablet report, sending...
I20260929 01:53:50.787560 21537 heartbeater.cc:507] Master 127.0.71.126:43657 requested a full tablet report, sending...
I20260929 01:53:50.787878 21536 heartbeater.cc:507] Master 127.0.71.124:43011 requested a full tablet report, sending...
I20260929 01:53:50.789379 20869 ts_manager.cc:194] Registered new tserver with Master: 88e3e35f4e6f4b46880366a101883381 (127.0.71.67:41347)
I20260929 01:53:50.789395 20958 ts_manager.cc:194] Registered new tserver with Master: 88e3e35f4e6f4b46880366a101883381 (127.0.71.67:41347)
I20260929 01:53:50.789386 21047 ts_manager.cc:194] Registered new tserver with Master: 88e3e35f4e6f4b46880366a101883381 (127.0.71.67:41347)
I20260929 01:53:50.790452 20869 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.67@KRBTEST.COM'} at 127.0.71.67:46047
I20260929 01:53:50.797032 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:53:50.804406 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:53:50.804621 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 20837
I20260929 01:53:50.805472 20899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:53:50.918450 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 20837
W20260929 01:53:50.939577 20942 connection.cc:573] server connection from 127.0.0.1:50468 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:53:50.940335 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:43657
--webserver_interface=127.0.71.126
--webserver_port=44749
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:38365,127.0.71.85:32921,127.0.71.86:40115,127.0.71.87:44833,127.0.71.88:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:51.252421 21395 heartbeater.cc:499] Master 127.0.71.126:43657 was elected leader, sending a full tablet report...
W20260929 01:53:51.254091 21395 heartbeater.cc:646] Failed to heartbeat to 127.0.71.126:43657 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.71.126:43657: connect: Connection refused (error 111)
W20260929 01:53:51.296870 21547 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:51.297204 21547 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:51.297358 21547 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:51.306151 21547 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:53:51.306284 21547 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:51.306418 21547 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:53:51.306511 21547 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:53:51.317883 21547 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38365,127.0.71.85:32921,127.0.71.86:40115,127.0.71.87:44833,127.0.71.88:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:43657
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=44749
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:51.319497 21547 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:51.320876 21547 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:51.329600 21554 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:51.329561 21553 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:51.329749 21556 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:51.330773 21547 server_base.cc:1061] running on GCE node
I20260929 01:53:51.331513 21547 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:51.333245 21547 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:51.334484 21547 hybrid_clock.cc:648] HybridClock initialized: now 1790646831334347 us; error 108 us; skew 500 ppm
I20260929 01:53:51.334888 21547 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:51 dist-test-slave-vz23 krb5kdc[20813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:51.338479 21547 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:53:51.340086 21547 webserver.cc:533] Webserver started at http://127.0.71.126:44749/ using document root <none> and password file <none>
I20260929 01:53:51.340917 21547 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:51.341053 21547 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:51.346508 21547 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:53:51.349331 21563 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:51.350590 21547 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:53:51.350770 21547 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "ce7ce17b42f04948b67de74b41285e11"
format_stamp: "Formatted at 2026-09-29 01:53:38 on dist-test-slave-vz23"
I20260929 01:53:51.351524 21547 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:51.368206 21547 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:51.371991 21547 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:51.372282 21547 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:51.393487 21547 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:43657
I20260929 01:53:51.393507 21614 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:43657 every 8 connection(s)
I20260929 01:53:51.395107 21547 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:53:51.402365 21615 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:53:51.403223 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 21547
I20260929 01:53:51.403513 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 21113
I20260929 01:53:51.406533 21615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: Bootstrap starting.
Sep 29 01:53:51 dist-test-slave-vz23 krb5kdc[20813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646826, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 01:53:51.431141 21244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:53:51.435767 21615 log.cc:826] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: Log is configured to *not* fsync() on all Append() calls
I20260929 01:53:51.445819 21615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:53:51.446542 21615 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: Bootstrap complete.
I20260929 01:53:51.460602 21615 raft_consensus.cc:359] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } }
I20260929 01:53:51.461902 21615 raft_consensus.cc:740] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: ce7ce17b42f04948b67de74b41285e11, State: Initialized, Role: FOLLOWER
I20260929 01:53:51.462901 21615 consensus_queue.cc:260] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } }
I20260929 01:53:51.464062 21621 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:51.464499 21621 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:51.465583 21615 sys_catalog.cc:565] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:53:51.486826 21632 catalog_manager.cc:1308] Loaded cluster ID: 1d32bc027e374184a8e577eeec0bb9fd
I20260929 01:53:51.487013 21632 catalog_manager.cc:1601] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: loading cluster ID for follower catalog manager: success
I20260929 01:53:51.491261 21632 catalog_manager.cc:1623] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: acquiring CA information for follower catalog manager: success
I20260929 01:53:51.494055 21632 catalog_manager.cc:1651] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Sep 29 01:53:51 dist-test-slave-vz23 krb5kdc[20813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646831, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:53:51.569897 21547 master_runner.cc:264] No leader master found from master ce7ce17b42f04948b67de74b41285e11
I20260929 01:53:51.570055 21547 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:53:51.585068 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 21113
I20260929 01:53:51.608703 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:36053
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--webserver_port=37987
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:38365,127.0.71.85:32921,127.0.71.86:40115,127.0.71.87:44833,127.0.71.88:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:51.792834 21537 heartbeater.cc:499] Master 127.0.71.126:43657 was elected leader, sending a full tablet report...
I20260929 01:53:51.803777 21580 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "88e3e35f4e6f4b46880366a101883381" instance_seqno: 1790646830739910) as {username='kudu', principal='kudu/127.0.71.67@KRBTEST.COM'} at 127.0.71.67:47703; Asking this server to re-register.
I20260929 01:53:51.804713 21537 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:51.806746 21580 ts_manager.cc:194] Registered new tserver with Master: 88e3e35f4e6f4b46880366a101883381 (127.0.71.67:41347)
I20260929 01:53:51.967448 21640 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce7ce17b42f04948b67de74b41285e11)
I20260929 01:53:51.967739 21640 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } }
I20260929 01:53:51.970034 21640 leader_election.cc:290] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ce7ce17b42f04948b67de74b41285e11 (127.0.71.126:43657), 444f97234e6d4c20be102cb7c0628408 (127.0.71.125:36483)
I20260929 01:53:51.970830 20968 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8cf5932291314506937aeaa8e67f3dff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "444f97234e6d4c20be102cb7c0628408" is_pre_election: true
I20260929 01:53:51.977355 21590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8cf5932291314506937aeaa8e67f3dff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce7ce17b42f04948b67de74b41285e11" is_pre_election: true
I20260929 01:53:51.977927 21590 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8cf5932291314506937aeaa8e67f3dff in term 2.
I20260929 01:53:51.978658 21032 leader_election.cc:304] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8cf5932291314506937aeaa8e67f3dff, ce7ce17b42f04948b67de74b41285e11; no voters: 444f97234e6d4c20be102cb7c0628408
I20260929 01:53:51.979220 21640 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 01:53:51.979364 21640 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [term 2 FOLLOWER]: Starting leader election (detected failure of leader ce7ce17b42f04948b67de74b41285e11)
I20260929 01:53:51.979498 21640 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:53:51.982090 21640 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } }
I20260929 01:53:51.982872 21640 leader_election.cc:290] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [CANDIDATE]: Term 3 election: Requested vote from peers ce7ce17b42f04948b67de74b41285e11 (127.0.71.126:43657), 444f97234e6d4c20be102cb7c0628408 (127.0.71.125:36483)
I20260929 01:53:51.983461 21590 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8cf5932291314506937aeaa8e67f3dff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce7ce17b42f04948b67de74b41285e11"
I20260929 01:53:51.983461 20968 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8cf5932291314506937aeaa8e67f3dff" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "444f97234e6d4c20be102cb7c0628408"
I20260929 01:53:51.983680 21590 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:53:51.987955 21590 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8cf5932291314506937aeaa8e67f3dff in term 3.
I20260929 01:53:51.988498 21032 leader_election.cc:304] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8cf5932291314506937aeaa8e67f3dff, ce7ce17b42f04948b67de74b41285e11; no voters: 444f97234e6d4c20be102cb7c0628408
I20260929 01:53:51.988862 21640 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [term 3 FOLLOWER]: Leader election won for term 3
I20260929 01:53:51.989395 21640 raft_consensus.cc:697] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [term 3 LEADER]: Becoming Leader. State: Replica: 8cf5932291314506937aeaa8e67f3dff, State: Running, Role: LEADER
I20260929 01:53:51.989840 21640 consensus_queue.cc:237] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } }
I20260929 01:53:51.991657 21642 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cf5932291314506937aeaa8e67f3dff. Latest consensus state: current_term: 3 leader_uuid: "8cf5932291314506937aeaa8e67f3dff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:51.992012 21642 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [sys.catalog]: This master's current role is: LEADER
I20260929 01:53:51.992803 21644 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:53:51.994915 21644 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:53:51.995796 21644 catalog_manager.cc:1308] Loaded cluster ID: 1d32bc027e374184a8e577eeec0bb9fd
I20260929 01:53:51.995906 21644 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:53:51.997329 21644 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:53:51.998339 21644 catalog_manager.cc:6230] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff: Loaded TSK: 0
I20260929 01:53:51.999269 21644 catalog_manager.cc:1563] Initializing in-progress tserver states...
W20260929 01:53:52.026906 21638 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:52.027205 21638 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:52.027313 21638 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:52.035948 21638 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:52.036145 21638 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:53:52.047134 21638 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38365,127.0.71.85:32921,127.0.71.86:40115,127.0.71.87:44833,127.0.71.88:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:36053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=37987
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:52.048846 21638 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:52.050133 21638 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:52.060112 21650 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:52.060245 21649 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:52.061494 21652 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:52.061666 21638 server_base.cc:1061] running on GCE node
I20260929 01:53:52.062354 21638 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:52.063566 21638 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:52.064801 21638 hybrid_clock.cc:648] HybridClock initialized: now 1790646832064649 us; error 122 us; skew 500 ppm
I20260929 01:53:52.065204 21638 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:52 dist-test-slave-vz23 krb5kdc[20813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:52.069005 21638 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:53:52.070888 21638 webserver.cc:533] Webserver started at http://127.0.71.65:37987/ using document root <none> and password file <none>
I20260929 01:53:52.071687 21638 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:52.071837 21638 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:52.077715 21638 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:53:52.081398 21659 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:52.083020 21638 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:53:52.083242 21638 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "ca2810190b0f4b078d7d671be7879084"
format_stamp: "Formatted at 2026-09-29 01:53:40 on dist-test-slave-vz23"
I20260929 01:53:52.084021 21638 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:52.111559 21638 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:52.115648 21638 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:52.115940 21638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:52.117038 21638 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:52.118606 21638 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:52.118731 21638 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:52.118863 21638 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:52.118950 21638 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:52.154331 21638 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:36053
I20260929 01:53:52.154419 21771 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:36053 every 8 connection(s)
I20260929 01:53:52.155881 21638 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:53:52.162874 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 21638
Sep 29 01:53:52 dist-test-slave-vz23 krb5kdc[20813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:53:52 dist-test-slave-vz23 krb5kdc[20813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:53:52 dist-test-slave-vz23 krb5kdc[20813](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:53:52.191720 21773 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43657
I20260929 01:53:52.192145 21773 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:52.193272 21773 heartbeater.cc:507] Master 127.0.71.126:43657 requested a full tablet report, sending...
I20260929 01:53:52.193310 21774 heartbeater.cc:344] Connected to a master server at 127.0.71.125:36483
I20260929 01:53:52.193459 21774 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:52.193995 21774 heartbeater.cc:507] Master 127.0.71.125:36483 requested a full tablet report, sending...
I20260929 01:53:52.195070 21580 ts_manager.cc:194] Registered new tserver with Master: ca2810190b0f4b078d7d671be7879084 (127.0.71.65:36053)
I20260929 01:53:52.195636 20958 ts_manager.cc:194] Re-registered known tserver with Master: ca2810190b0f4b078d7d671be7879084 (127.0.71.65:36053)
I20260929 01:53:52.198088 21772 heartbeater.cc:344] Connected to a master server at 127.0.71.124:43011
I20260929 01:53:52.198218 21772 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:52.198678 21772 heartbeater.cc:507] Master 127.0.71.124:43011 requested a full tablet report, sending...
I20260929 01:53:52.199795 21047 ts_manager.cc:194] Re-registered known tserver with Master: ca2810190b0f4b078d7d671be7879084 (127.0.71.65:36053)
I20260929 01:53:52.202210 21047 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:46825
I20260929 01:53:52.210273 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 21638
W20260929 01:53:52.211279 21784 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:52.208223 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:35836 (local address 127.0.71.65:36053)
0929 01:53:52.208835 (+ 612us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:52.208841 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:52.208869 (+ 28us) server_negotiation.cc:408] Connection header received
0929 01:53:52.208998 (+ 129us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:53:52.209004 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:53:52.209184 (+ 180us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:53:52.209328 (+ 144us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:53:52.210130 (+ 802us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35836: BlockingRecv error: recv got EOF from 127.0.0.1:35836 (error 108)
Metrics: {"server-negotiator.queue_time_us":471,"thread_start_us":318,"threads_started":1}
W20260929 01:53:52.212610 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 21638: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:36053: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:53:52.212738 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 21638
I20260929 01:53:52.231067 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:36053
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--webserver_port=37987
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:38365,127.0.71.85:32921,127.0.71.86:40115,127.0.71.87:44833,127.0.71.88:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:52.261040 21395 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43657
I20260929 01:53:52.261261 21395 heartbeater.cc:499] Master 127.0.71.126:43657 was elected leader, sending a full tablet report...
I20260929 01:53:52.262208 21580 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d39478ed416a473d8959c68989748cc3" instance_seqno: 1790646830147555) as {username='kudu', principal='kudu/127.0.71.66@KRBTEST.COM'} at 127.0.71.66:34777; Asking this server to re-register.
I20260929 01:53:52.262838 21395 heartbeater.cc:461] Registering TS with master...
I20260929 01:53:52.263903 21580 ts_manager.cc:194] Registered new tserver with Master: d39478ed416a473d8959c68989748cc3 (127.0.71.66:46353)
I20260929 01:53:52.420261 21788 raft_consensus.cc:493] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader ce7ce17b42f04948b67de74b41285e11)
I20260929 01:53:52.420529 21788 raft_consensus.cc:515] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } }
I20260929 01:53:52.422381 21788 leader_election.cc:290] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers ce7ce17b42f04948b67de74b41285e11 (127.0.71.126:43657), 8cf5932291314506937aeaa8e67f3dff (127.0.71.124:43011)
I20260929 01:53:52.427522 21590 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [term 3 FOLLOWER]: Refusing update from remote peer 8cf5932291314506937aeaa8e67f3dff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 01:53:52.428650 21642 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [LEADER]: Connected to new peer: Peer: permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:53:52.432745 21058 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "444f97234e6d4c20be102cb7c0628408" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "8cf5932291314506937aeaa8e67f3dff" is_pre_election: true
I20260929 01:53:52.432998 21589 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "444f97234e6d4c20be102cb7c0628408" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ce7ce17b42f04948b67de74b41285e11" is_pre_election: true
I20260929 01:53:52.433961 20942 leader_election.cc:304] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 444f97234e6d4c20be102cb7c0628408; no voters: 8cf5932291314506937aeaa8e67f3dff, ce7ce17b42f04948b67de74b41285e11
I20260929 01:53:52.434608 21788 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [term 2 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 01:53:52.436239 21791 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cf5932291314506937aeaa8e67f3dff. Latest consensus state: current_term: 3 leader_uuid: "8cf5932291314506937aeaa8e67f3dff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:52.436683 21791 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:52.437762 21640 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "8cf5932291314506937aeaa8e67f3dff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:52.438115 21640 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [sys.catalog]: This master's current role is: LEADER
I20260929 01:53:52.438920 20968 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:53:52.441407 20968 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [term 3 FOLLOWER]: Refusing update from remote peer 8cf5932291314506937aeaa8e67f3dff: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 01:53:52.442855 21794 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [LEADER]: Connected to new peer: Peer: permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:53:52.447957 21791 sys_catalog.cc:455] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "8cf5932291314506937aeaa8e67f3dff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:52.448227 21791 sys_catalog.cc:458] T 00000000000000000000000000000000 P ce7ce17b42f04948b67de74b41285e11 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:52.450081 21788 sys_catalog.cc:455] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 8cf5932291314506937aeaa8e67f3dff. Latest consensus state: current_term: 3 leader_uuid: "8cf5932291314506937aeaa8e67f3dff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:52.450302 21788 sys_catalog.cc:458] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:53:52.450028 21798 sys_catalog.cc:455] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "8cf5932291314506937aeaa8e67f3dff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:52.450495 21798 sys_catalog.cc:458] T 00000000000000000000000000000000 P 444f97234e6d4c20be102cb7c0628408 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:53:52.588747 21786 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:52.589102 21786 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:52.589283 21786 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:53:52.589380 21786 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:52.598198 21786 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:52.598474 21786 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:53:52.609733 21786 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38365,127.0.71.85:32921,127.0.71.86:40115,127.0.71.87:44833,127.0.71.88:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:36053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=37987
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:52.611450 21786 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:52.612964 21786 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:52.623286 21804 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:52.623311 21806 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:52.623692 21803 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:52.624632 21786 server_base.cc:1061] running on GCE node
I20260929 01:53:52.625216 21786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:52.626354 21786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:52.627663 21786 hybrid_clock.cc:648] HybridClock initialized: now 1790646832627517 us; error 102 us; skew 500 ppm
I20260929 01:53:52.627981 21786 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:52 dist-test-slave-vz23 krb5kdc[20813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646832, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:52.631739 21786 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:53:52.633687 21786 webserver.cc:533] Webserver started at http://127.0.71.65:37987/ using document root <none> and password file <none>
I20260929 01:53:52.634393 21786 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:52.634476 21786 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:52.639931 21786 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:53:52.643237 21813 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:52.644644 21786 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:53:52.644800 21786 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "ca2810190b0f4b078d7d671be7879084"
format_stamp: "Formatted at 2026-09-29 01:53:40 on dist-test-slave-vz23"
I20260929 01:53:52.645476 21786 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:52.663545 21786 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:52.667479 21786 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:52.667714 21786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:52.668654 21786 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:52.670367 21786 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:52.670440 21786 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:52.670526 21786 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:52.670569 21786 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:53:52.670897 21786 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:53:52.670998 21786 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:53:52.671087 21786 fault_injection.cc:59] injecting fault for kudu (pid 21786): FLAGS_fault_before_start (process will exit)
I20260929 01:53:52.694792 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:36053
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--webserver_port=37987
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:38365,127.0.71.85:32921,127.0.71.86:40115,127.0.71.87:44833,127.0.71.88:33171
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:53:52.960193 21822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "8cf5932291314506937aeaa8e67f3dff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "ce7ce17b42f04948b67de74b41285e11" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43657 } } peers { permanent_uuid: "444f97234e6d4c20be102cb7c0628408" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36483 } } peers { permanent_uuid: "8cf5932291314506937aeaa8e67f3dff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43011 } } }
I20260929 01:53:52.960541 21822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8cf5932291314506937aeaa8e67f3dff [sys.catalog]: This master's current role is: LEADER
W20260929 01:53:53.045264 21821 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:53:53.045608 21821 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:53:53.045763 21821 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:53:53.054225 21821 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:53:53.054474 21821 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:53:53.065368 21821 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38365,127.0.71.85:32921,127.0.71.86:40115,127.0.71.87:44833,127.0.71.88:33171
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:36053
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=37987
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:43657,127.0.71.125:36483,127.0.71.124:43011
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:53:53.067185 21821 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:53:53.068572 21821 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:53:53.078970 21831 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:53.079223 21830 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:53:53.079938 21833 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:53:53.080281 21821 server_base.cc:1061] running on GCE node
I20260929 01:53:53.081003 21821 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:53:53.082382 21821 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:53:53.083618 21821 hybrid_clock.cc:648] HybridClock initialized: now 1790646833083472 us; error 117 us; skew 500 ppm
I20260929 01:53:53.083967 21821 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:53:53 dist-test-slave-vz23 krb5kdc[20813](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646833, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:53:53.087740 21821 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:53:53.089534 21821 webserver.cc:533] Webserver started at http://127.0.71.65:37987/ using document root <none> and password file <none>
I20260929 01:53:53.090210 21821 fs_manager.cc:362] Metadata directory not provided
I20260929 01:53:53.090318 21821 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:53:53.095855 21821 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:53:53.098660 21840 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:53.100033 21821 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 01:53:53.100198 21821 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "ca2810190b0f4b078d7d671be7879084"
format_stamp: "Formatted at 2026-09-29 01:53:40 on dist-test-slave-vz23"
I20260929 01:53:53.100950 21821 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:53:53.119009 21821 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:53:53.123303 21821 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:53:53.123548 21821 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:53:53.124547 21821 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:53:53.126263 21821 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:53:53.126363 21821 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:53.126443 21821 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:53:53.126499 21821 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:53:53.164631 21821 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:36053
I20260929 01:53:53.164673 21952 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:36053 every 8 connection(s)
I20260929 01:53:53.166142 21821 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:53:53.167078 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 21821
I20260929 01:53:53.167421 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 21821
W20260929 01:53:53.188764 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 21821: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:36053: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:53:53.188915 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 21821
I20260929 01:53:53.206107 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 21263
W20260929 01:53:53.206777 21959 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:53.170381 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:43775 (local address 127.0.71.125:36483)
0929 01:53:53.184337 (+ 13956us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:53.184342 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:53.205863 (+ 21521us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:43775: BlockingRecv error: recv got EOF from 127.0.71.65:43775 (error 108)
Metrics: {"server-negotiator.queue_time_us":13837,"thread_start_us":13675,"threads_started":1}
W20260929 01:53:53.206818 21962 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:53.170923 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:58855 (local address 127.0.71.124:43011)
0929 01:53:53.190882 (+ 19959us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:53.190887 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:53.190907 (+ 20us) server_negotiation.cc:408] Connection header received
0929 01:53:53.190957 (+ 50us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:53:53.190961 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:53:53.191019 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:53:53.191098 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:53:53.205896 (+ 14798us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:58855: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":19838,"thread_start_us":19695,"threads_started":1}
W20260929 01:53:53.207374 21963 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:53.171365 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:46629 (local address 127.0.71.126:43657)
0929 01:53:53.190823 (+ 19458us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:53.190828 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:53.205872 (+ 15044us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:46629: BlockingRecv error: recv got EOF from 127.0.71.65:46629 (error 108)
Metrics: {"server-negotiator.queue_time_us":19366,"thread_start_us":20653,"threads_started":1}
W20260929 01:53:53.209446 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 21263: Not authorized: Client connection negotiation failed: client connection to 127.0.71.66:46353: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:53:53.209607 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 21263
I20260929 01:53:53.225945 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 21408
W20260929 01:53:53.228716 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 21408: Not authorized: Client connection negotiation failed: client connection to 127.0.71.67:41347: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:53:53.228861 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 21408
I20260929 01:53:53.244863 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 21547
W20260929 01:53:53.246896 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 21547: Not authorized: Client connection negotiation failed: client connection to 127.0.71.126:43657: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:53:53.247072 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 21547
W20260929 01:53:53.247104 21963 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:53:53.245782 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:40820 (local address 127.0.71.126:43657)
0929 01:53:53.245981 (+ 199us) server_negotiation.cc:207] Beginning negotiation
0929 01:53:53.245985 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0929 01:53:53.246002 (+ 17us) server_negotiation.cc:408] Connection header received
0929 01:53:53.246063 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:53:53.246068 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:53:53.246127 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:53:53.246214 (+ 87us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:53:53.246880 (+ 666us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40820: BlockingRecv error: recv got EOF from 127.0.0.1:40820 (error 108)
Metrics: {"server-negotiator.queue_time_us":65}
I20260929 01:53:53.263629 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 20926
I20260929 01:53:53.264843 20988 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:53:53.377964 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 20926
I20260929 01:53:53.399221 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 21016
I20260929 01:53:53.400394 21078 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:53:53.514125 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 21016
2026-09-29T01:53:53Z chronyd exiting
2026-09-29T01:53:53Z chronyd exiting
2026-09-29T01:53:53Z chronyd exiting
2026-09-29T01:53:53Z chronyd exiting
2026-09-29T01:53:53Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestBasicOperation/8 (18056 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/9
Loading random data
Initializing database '/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:53:53 dist-test-slave-vz23 krb5kdc[21972](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:53:53 dist-test-slave-vz23 krb5kdc[21972](info): set up 2 sockets
Sep 29 01:53:53 dist-test-slave-vz23 krb5kdc[21972](info): commencing operation
krb5kdc: starting...
W20260929 01:53:55.677839 285 mini_kdc.cc:121] Time spent starting KDC: real 2.075s user 0.000s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 01:53:55 dist-test-slave-vz23 krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for hive/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "hive/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/hive_127.0.0.1.keytab.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/hive_127.0.0.1.keytab.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5.conf
2026-09-29 01:54:03: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
Sep 29 01:54:05 dist-test-slave-vz23 krb5kdc[21972](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790646845, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 01:54:11 dist-test-slave-vz23 krb5kdc[21972](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790646851, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 01:54:11 dist-test-slave-vz23 krb5kdc[21972](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790646851, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:54:11.912516 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:36771
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:54:12.249652 22265 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:54:12.249987 22265 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:54:12.250080 22265 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:54:12.258662 22265 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:54:12.258775 22265 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:54:12.258834 22265 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:54:12.258874 22265 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:54:12.269876 22265 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:36771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:54:12.271452 22265 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:54:12.272742 22265 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:54:12.280931 22270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:12.280956 22271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:12.281880 22273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:54:12.282276 22265 server_base.cc:1061] running on GCE node
I20260929 01:54:12.283134 22265 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:54:12.283316 22265 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:54:12.283419 22265 hybrid_clock.cc:648] HybridClock initialized: now 1790646852283380 us; error 0 us; skew 500 ppm
I20260929 01:54:12.283751 22265 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:54:12 dist-test-slave-vz23 krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:54:12.287221 22265 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:54:12.289252 22265 webserver.cc:533] Webserver started at http://127.0.71.126:34037/ using document root <none> and password file <none>
I20260929 01:54:12.290037 22265 fs_manager.cc:362] Metadata directory not provided
I20260929 01:54:12.290182 22265 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:54:12.290541 22265 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:54:12.293015 22265 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "7f6befd994534bbebdb63da555aabb73"
format_stamp: "Formatted at 2026-09-29 01:54:12 on dist-test-slave-vz23"
I20260929 01:54:12.293848 22265 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "7f6befd994534bbebdb63da555aabb73"
format_stamp: "Formatted at 2026-09-29 01:54:12 on dist-test-slave-vz23"
I20260929 01:54:12.299211 22265 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 01:54:12.302765 22279 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:12.304152 22265 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:54:12.304327 22265 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "7f6befd994534bbebdb63da555aabb73"
format_stamp: "Formatted at 2026-09-29 01:54:12 on dist-test-slave-vz23"
I20260929 01:54:12.304567 22265 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:54:12.329187 22265 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:54:12.338479 22265 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:54:12.338786 22265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:54:12.360116 22265 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:36771
I20260929 01:54:12.360143 22330 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:36771 every 8 connection(s)
I20260929 01:54:12.361725 22265 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:54:12.365296 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 22265
I20260929 01:54:12.365502 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:54:12.366529 22331 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:54:12.375149 22331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.71.125@KRBTEST.COM; defaulting to no policy
Sep 29 01:54:12 dist-test-slave-vz23 krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.125@KRBTEST.COM" created.
I20260929 01:54:12.398264 22331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:54:12.399479 22282 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.125:46761: connect: Connection refused (error 111)
W20260929 01:54:12.400985 22331 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:46761: Network error: Client connection negotiation failed: client connection to 127.0.71.125:46761: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.125@KRBTEST.COM" created.
I20260929 01:54:12.450338 22331 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } attempt: 1
W20260929 01:54:12.452538 22331 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:46761: Network error: Client connection negotiation failed: client connection to 127.0.71.125:46761: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:54:12.465682 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:46761
--webserver_interface=127.0.71.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:54:12.520850 22331 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } attempt: 2
W20260929 01:54:12.522921 22331 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:46761: Network error: Client connection negotiation failed: client connection to 127.0.71.125:46761: connect: Connection refused (error 111)
I20260929 01:54:12.614215 22331 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } attempt: 3
W20260929 01:54:12.616313 22331 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:46761: Network error: Client connection negotiation failed: client connection to 127.0.71.125:46761: connect: Connection refused (error 111)
I20260929 01:54:12.759564 22331 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } attempt: 4
W20260929 01:54:12.761765 22331 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:46761: Network error: Client connection negotiation failed: client connection to 127.0.71.125:46761: connect: Connection refused (error 111)
W20260929 01:54:12.801695 22340 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:54:12.802062 22340 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:54:12.802212 22340 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:54:12.810587 22340 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:54:12.810730 22340 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:54:12.810847 22340 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:54:12.810926 22340 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:54:12.822345 22340 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:46761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:54:12.824005 22340 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:54:12.825369 22340 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:54:12.834203 22346 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:12.834262 22348 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:12.834257 22345 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:54:12.835157 22340 server_base.cc:1061] running on GCE node
I20260929 01:54:12.835954 22340 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:54:12.836151 22340 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:54:12.836256 22340 hybrid_clock.cc:648] HybridClock initialized: now 1790646852836231 us; error 0 us; skew 500 ppm
I20260929 01:54:12.836663 22340 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:54:12 dist-test-slave-vz23 krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:54:12.840060 22340 init.cc:377] Logged in from keytab as kudu/127.0.71.125@KRBTEST.COM (short username kudu)
I20260929 01:54:12.842131 22340 webserver.cc:533] Webserver started at http://127.0.71.125:36183/ using document root <none> and password file <none>
I20260929 01:54:12.842857 22340 fs_manager.cc:362] Metadata directory not provided
I20260929 01:54:12.842993 22340 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:54:12.843328 22340 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:54:12.845798 22340 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/data/instance:
uuid: "ff24fc03c4e74eae9c1472eaa7875378"
format_stamp: "Formatted at 2026-09-29 01:54:12 on dist-test-slave-vz23"
I20260929 01:54:12.846586 22340 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/wal/instance:
uuid: "ff24fc03c4e74eae9c1472eaa7875378"
format_stamp: "Formatted at 2026-09-29 01:54:12 on dist-test-slave-vz23"
I20260929 01:54:12.852029 22340 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 01:54:12.855646 22354 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:12.857296 22340 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:54:12.857496 22340 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "ff24fc03c4e74eae9c1472eaa7875378"
format_stamp: "Formatted at 2026-09-29 01:54:12 on dist-test-slave-vz23"
I20260929 01:54:12.857693 22340 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:54:12.872200 22340 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:54:12.876410 22340 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:54:12.876758 22340 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:54:12.897950 22340 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:46761
I20260929 01:54:12.897974 22405 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:46761 every 8 connection(s)
I20260929 01:54:12.899511 22340 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:54:12.903925 22406 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:54:12.907174 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 22340
I20260929 01:54:12.907372 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:54:12.911772 22406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:54:12 dist-test-slave-vz23 krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.124@KRBTEST.COM" created.
I20260929 01:54:12.935556 22406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:54:12 dist-test-slave-vz23 krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:54:12.952874 22406 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:54:12.954228 22358 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.124:45059: connect: Connection refused (error 111)
W20260929 01:54:12.956178 22406 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:45059: Network error: Client connection negotiation failed: client connection to 127.0.71.124:45059: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.124@KRBTEST.COM" created.
I20260929 01:54:13.005605 22406 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } attempt: 1
W20260929 01:54:13.008215 22406 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:45059: Network error: Client connection negotiation failed: client connection to 127.0.71.124:45059: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:54:13.015607 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:45059
--webserver_interface=127.0.71.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:54:13.061005 22331 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } attempt: 5
Sep 29 01:54:13 dist-test-slave-vz23 krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:54:13.074765 22331 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:54:13.076437 22406 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } attempt: 2
W20260929 01:54:13.076817 22331 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:45059: Network error: Client connection negotiation failed: client connection to 127.0.71.124:45059: connect: Connection refused (error 111)
W20260929 01:54:13.078351 22406 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:45059: Network error: Client connection negotiation failed: client connection to 127.0.71.124:45059: connect: Connection refused (error 111)
I20260929 01:54:13.128082 22331 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } attempt: 1
W20260929 01:54:13.130250 22331 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:45059: Network error: Client connection negotiation failed: client connection to 127.0.71.124:45059: connect: Connection refused (error 111)
I20260929 01:54:13.169610 22406 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } attempt: 3
W20260929 01:54:13.171856 22406 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:45059: Network error: Client connection negotiation failed: client connection to 127.0.71.124:45059: connect: Connection refused (error 111)
I20260929 01:54:13.198561 22331 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } attempt: 2
W20260929 01:54:13.200627 22331 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:45059: Network error: Client connection negotiation failed: client connection to 127.0.71.124:45059: connect: Connection refused (error 111)
I20260929 01:54:13.306936 22331 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } attempt: 3
W20260929 01:54:13.309087 22331 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:45059: Network error: Client connection negotiation failed: client connection to 127.0.71.124:45059: connect: Connection refused (error 111)
I20260929 01:54:13.315115 22406 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } attempt: 4
W20260929 01:54:13.317106 22406 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:45059: Network error: Client connection negotiation failed: client connection to 127.0.71.124:45059: connect: Connection refused (error 111)
W20260929 01:54:13.366915 22416 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:54:13.367205 22416 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:54:13.367302 22416 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:54:13.375717 22416 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:54:13.375820 22416 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:54:13.375874 22416 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:54:13.375923 22416 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:54:13.386763 22416 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:45059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:54:13.388293 22416 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:54:13.389642 22416 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:54:13.398219 22422 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:13.398376 22421 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:13.398880 22424 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:54:13.399160 22416 server_base.cc:1061] running on GCE node
I20260929 01:54:13.399921 22416 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:54:13.400086 22416 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:54:13.400173 22416 hybrid_clock.cc:648] HybridClock initialized: now 1790646853400152 us; error 0 us; skew 500 ppm
I20260929 01:54:13.400542 22416 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:54:13 dist-test-slave-vz23 krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:54:13.404022 22416 init.cc:377] Logged in from keytab as kudu/127.0.71.124@KRBTEST.COM (short username kudu)
I20260929 01:54:13.406035 22416 webserver.cc:533] Webserver started at http://127.0.71.124:37481/ using document root <none> and password file <none>
I20260929 01:54:13.406724 22416 fs_manager.cc:362] Metadata directory not provided
I20260929 01:54:13.406836 22416 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:54:13.407181 22416 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:54:13.409677 22416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/data/instance:
uuid: "84706cb3b1a64ebc9710b0c330b6ea73"
format_stamp: "Formatted at 2026-09-29 01:54:13 on dist-test-slave-vz23"
I20260929 01:54:13.410408 22416 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/wal/instance:
uuid: "84706cb3b1a64ebc9710b0c330b6ea73"
format_stamp: "Formatted at 2026-09-29 01:54:13 on dist-test-slave-vz23"
I20260929 01:54:13.415511 22416 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 01:54:13.418780 22430 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:13.420091 22416 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:54:13.420293 22416 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "84706cb3b1a64ebc9710b0c330b6ea73"
format_stamp: "Formatted at 2026-09-29 01:54:13 on dist-test-slave-vz23"
I20260929 01:54:13.420523 22416 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:54:13.472103 22416 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:54:13.476289 22416 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:54:13.476608 22416 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:54:13.486344 22331 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } attempt: 4
W20260929 01:54:13.488798 22331 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:45059: Network error: Client connection negotiation failed: client connection to 127.0.71.124:45059: connect: Connection refused (error 111)
I20260929 01:54:13.498027 22416 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:45059
I20260929 01:54:13.498031 22481 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:45059 every 8 connection(s)
I20260929 01:54:13.499542 22416 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:54:13.503814 22482 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:54:13.507812 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 22416
I20260929 01:54:13.508028 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:54:13.511051 22482 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:54:13 dist-test-slave-vz23 krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:54:13 dist-test-slave-vz23 krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:54:13.532866 22482 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:54:13 dist-test-slave-vz23 krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:54:13 dist-test-slave-vz23 krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:54:13.549207 22482 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:54:13 dist-test-slave-vz23 krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:54:13 dist-test-slave-vz23 krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646853, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:54:13.568121 22482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73: Bootstrap starting.
I20260929 01:54:13.571313 22482 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73: Neither blocks nor log segments found. Creating new log.
I20260929 01:54:13.572638 22482 log.cc:826] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73: Log is configured to *not* fsync() on all Append() calls
I20260929 01:54:13.575104 22482 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73: No bootstrap required, opened a new log
I20260929 01:54:13.580569 22482 raft_consensus.cc:359] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } }
I20260929 01:54:13.580871 22482 raft_consensus.cc:385] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:54:13.581022 22482 raft_consensus.cc:740] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84706cb3b1a64ebc9710b0c330b6ea73, State: Initialized, Role: FOLLOWER
I20260929 01:54:13.581642 22482 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } }
I20260929 01:54:13.582679 22492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:13.583039 22492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:54:13.583695 22482 sys_catalog.cc:565] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:54:13.597234 22503 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:54:13.597373 22503 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:54:13.592907 22416 master_runner.cc:187] Error getting master registration for 127.0.71.126:36771: OK, instance_id { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" instance_seqno: 1790646852354407 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:54:13.599045 22416 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:54:13.616369 22406 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } attempt: 5
Sep 29 01:54:13 dist-test-slave-vz23 krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:54:13.637370 22406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378: Bootstrap starting.
I20260929 01:54:13.641749 22406 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378: Neither blocks nor log segments found. Creating new log.
I20260929 01:54:13.643137 22406 log.cc:826] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378: Log is configured to *not* fsync() on all Append() calls
I20260929 01:54:13.646821 22406 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378: No bootstrap required, opened a new log
I20260929 01:54:13.651860 22406 raft_consensus.cc:359] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } }
I20260929 01:54:13.652159 22406 raft_consensus.cc:385] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:54:13.652304 22406 raft_consensus.cc:740] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff24fc03c4e74eae9c1472eaa7875378, State: Initialized, Role: FOLLOWER
I20260929 01:54:13.652994 22406 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } }
I20260929 01:54:13.653968 22506 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:13.654266 22506 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:54:13.654875 22406 sys_catalog.cc:565] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:54:13.666461 22517 catalog_manager.cc:1607] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:54:13.666663 22517 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:54:13.663331 22340 master_runner.cc:187] Error getting master registration for 127.0.71.126:36771: OK, instance_id { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" instance_seqno: 1790646852354407 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:54:13.670680 22340 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:54:13.700814 22506 raft_consensus.cc:493] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:54:13.701110 22506 raft_consensus.cc:515] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } }
I20260929 01:54:13.702998 22506 leader_election.cc:290] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7f6befd994534bbebdb63da555aabb73 (127.0.71.126:36771), 84706cb3b1a64ebc9710b0c330b6ea73 (127.0.71.124:45059)
I20260929 01:54:13.703940 22457 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ff24fc03c4e74eae9c1472eaa7875378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" is_pre_election: true
I20260929 01:54:13.704520 22457 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff24fc03c4e74eae9c1472eaa7875378 in term 0.
I20260929 01:54:13.705158 22358 leader_election.cc:304] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84706cb3b1a64ebc9710b0c330b6ea73, ff24fc03c4e74eae9c1472eaa7875378; no voters:
I20260929 01:54:13.705579 22506 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:54:13.705782 22506 raft_consensus.cc:493] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:54:13.705900 22506 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:54:13.709427 22506 raft_consensus.cc:515] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } }
I20260929 01:54:13.710213 22506 leader_election.cc:290] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [CANDIDATE]: Term 1 election: Requested vote from peers 7f6befd994534bbebdb63da555aabb73 (127.0.71.126:36771), 84706cb3b1a64ebc9710b0c330b6ea73 (127.0.71.124:45059)
I20260929 01:54:13.711169 22457 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ff24fc03c4e74eae9c1472eaa7875378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "84706cb3b1a64ebc9710b0c330b6ea73"
I20260929 01:54:13.711372 22457 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:54:13.714988 22457 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff24fc03c4e74eae9c1472eaa7875378 in term 1.
I20260929 01:54:13.717388 22358 leader_election.cc:304] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 84706cb3b1a64ebc9710b0c330b6ea73, ff24fc03c4e74eae9c1472eaa7875378; no voters:
I20260929 01:54:13.717739 22506 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:54:13.718375 22506 raft_consensus.cc:697] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [term 1 LEADER]: Becoming Leader. State: Replica: ff24fc03c4e74eae9c1472eaa7875378, State: Running, Role: LEADER
I20260929 01:54:13.718940 22506 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } }
W20260929 01:54:13.728752 22436 tablet.cc:2433] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:54:13.731386 22519 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff24fc03c4e74eae9c1472eaa7875378. Latest consensus state: current_term: 1 leader_uuid: "ff24fc03c4e74eae9c1472eaa7875378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:13.732170 22519 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [sys.catalog]: This master's current role is: LEADER
I20260929 01:54:13.733495 22521 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:54:13.737021 22521 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:54:13.748875 22457 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [term 1 FOLLOWER]: Refusing update from remote peer ff24fc03c4e74eae9c1472eaa7875378: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:54:13.750175 22519 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:54:13.765976 22331 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } attempt: 5
I20260929 01:54:13.771003 22492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff24fc03c4e74eae9c1472eaa7875378. Latest consensus state: current_term: 1 leader_uuid: "ff24fc03c4e74eae9c1472eaa7875378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:13.771692 22492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:54:13.776245 22522 mvcc.cc:204] Tried to move back new op lower bound from 7334489512942022656 to 7334489512838905856. Current Snapshot: MvccSnapshot[applied={T|T < 7334489512942022656}]
Sep 29 01:54:13 dist-test-slave-vz23 krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646852, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:54:13.786654 22506 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff24fc03c4e74eae9c1472eaa7875378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:13.786963 22506 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [sys.catalog]: This master's current role is: LEADER
I20260929 01:54:13.787830 22521 catalog_manager.cc:1396] Generated new cluster ID: 4b631c8567cf41bab15ecc52fd208f7f
I20260929 01:54:13.787930 22521 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:54:13.805824 22523 mvcc.cc:204] Tried to move back new op lower bound from 7334489512942022656 to 7334489512838905856. Current Snapshot: MvccSnapshot[applied={T|T < 7334489512942022656}]
I20260929 01:54:13.811892 22331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: Bootstrap starting.
I20260929 01:54:13.815104 22492 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff24fc03c4e74eae9c1472eaa7875378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:13.815323 22492 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:54:13.819209 22331 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: Neither blocks nor log segments found. Creating new log.
I20260929 01:54:13.821463 22331 log.cc:826] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: Log is configured to *not* fsync() on all Append() calls
I20260929 01:54:13.834501 22521 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 01:54:13.836344 22521 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:54:13.839994 22331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: No bootstrap required, opened a new log
I20260929 01:54:13.847594 22331 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } }
I20260929 01:54:13.847941 22331 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:54:13.848063 22331 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6befd994534bbebdb63da555aabb73, State: Initialized, Role: FOLLOWER
I20260929 01:54:13.857090 22521 catalog_manager.cc:6219] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378: Generated new TSK 0
I20260929 01:54:13.858265 22521 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:54:13.860903 22331 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } }
I20260929 01:54:13.862176 22528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:13.862673 22528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:54:13.863508 22331 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:54:13.863540 22306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ff24fc03c4e74eae9c1472eaa7875378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6befd994534bbebdb63da555aabb73" is_pre_election: true
I20260929 01:54:13.864074 22306 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff24fc03c4e74eae9c1472eaa7875378 in term 0.
I20260929 01:54:13.864966 22305 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ff24fc03c4e74eae9c1472eaa7875378" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7f6befd994534bbebdb63da555aabb73"
I20260929 01:54:13.865237 22305 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:54:13.868532 22305 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff24fc03c4e74eae9c1472eaa7875378 in term 1.
I20260929 01:54:13.870100 22304 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [term 1 FOLLOWER]: Refusing update from remote peer ff24fc03c4e74eae9c1472eaa7875378: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:54:13.871122 22519 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
I20260929 01:54:13.928081 22536 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "ff24fc03c4e74eae9c1472eaa7875378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:13.928473 22536 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:54:13.928783 22528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff24fc03c4e74eae9c1472eaa7875378. Latest consensus state: current_term: 1 leader_uuid: "ff24fc03c4e74eae9c1472eaa7875378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:13.928910 22528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [sys.catalog]: This master's current role is: FOLLOWER
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.65@KRBTEST.COM" created.
I20260929 01:54:13.953559 22546 catalog_manager.cc:818] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
W20260929 01:54:13.975145 285 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.0.71.65: real 0.105s user 0.001s sys 0.019s
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:54:14.024922 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:54:14.261758 22519 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff24fc03c4e74eae9c1472eaa7875378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:14.262022 22519 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [sys.catalog]: This master's current role is: LEADER
W20260929 01:54:14.388836 22549 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:54:14.389156 22549 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:54:14.389245 22549 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:54:14.397991 22549 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:54:14.398257 22549 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:54:14.409471 22549 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:54:14.411018 22549 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:54:14.412300 22549 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:54:14.422775 22555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:14.423008 22556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:14.422982 22558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:54:14.423736 22549 server_base.cc:1061] running on GCE node
I20260929 01:54:14.424317 22549 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:54:14.424511 22549 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:54:14.424624 22549 hybrid_clock.cc:648] HybridClock initialized: now 1790646854424582 us; error 0 us; skew 500 ppm
I20260929 01:54:14.424935 22549 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:54:14 dist-test-slave-vz23 krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:54:14.428254 22549 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:54:14.429955 22549 webserver.cc:533] Webserver started at http://127.0.71.65:34679/ using document root <none> and password file <none>
I20260929 01:54:14.430686 22549 fs_manager.cc:362] Metadata directory not provided
I20260929 01:54:14.430776 22549 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:54:14.431056 22549 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:54:14.433396 22549 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "afc2290d563b4743b87fac3c7c266de6"
format_stamp: "Formatted at 2026-09-29 01:54:14 on dist-test-slave-vz23"
I20260929 01:54:14.434083 22549 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "afc2290d563b4743b87fac3c7c266de6"
format_stamp: "Formatted at 2026-09-29 01:54:14 on dist-test-slave-vz23"
I20260929 01:54:14.439489 22549 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260929 01:54:14.442761 22564 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:14.444126 22549 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 01:54:14.444285 22549 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "afc2290d563b4743b87fac3c7c266de6"
format_stamp: "Formatted at 2026-09-29 01:54:14 on dist-test-slave-vz23"
I20260929 01:54:14.444491 22549 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:54:14.469741 22549 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:54:14.473987 22549 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:54:14.474375 22549 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:54:14.475333 22549 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:54:14.476987 22549 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:54:14.477097 22549 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:14.477248 22549 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:54:14.477320 22549 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:14.516522 22549 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:42833
I20260929 01:54:14.516587 22676 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:42833 every 8 connection(s)
I20260929 01:54:14.517946 22549 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:54:14.519851 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 22549
I20260929 01:54:14.520059 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
Sep 29 01:54:14 dist-test-slave-vz23 krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:54:14 dist-test-slave-vz23 krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:54:14 dist-test-slave-vz23 krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646854, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:54:14.554617 22677 heartbeater.cc:344] Connected to a master server at 127.0.71.124:45059
I20260929 01:54:14.555017 22677 heartbeater.cc:461] Registering TS with master...
I20260929 01:54:14.555933 22678 heartbeater.cc:344] Connected to a master server at 127.0.71.126:36771
WARNING: no policy specified for kudu/127.0.71.66@KRBTEST.COM; defaulting to no policy
I20260929 01:54:14.556092 22678 heartbeater.cc:461] Registering TS with master...
I20260929 01:54:14.556217 22677 heartbeater.cc:507] Master 127.0.71.124:45059 requested a full tablet report, sending...
I20260929 01:54:14.556660 22678 heartbeater.cc:507] Master 127.0.71.126:36771 requested a full tablet report, sending...
I20260929 01:54:14.557545 22679 heartbeater.cc:344] Connected to a master server at 127.0.71.125:46761
I20260929 01:54:14.557691 22679 heartbeater.cc:461] Registering TS with master...
I20260929 01:54:14.558188 22679 heartbeater.cc:507] Master 127.0.71.125:46761 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.66@KRBTEST.COM" created.
I20260929 01:54:14.559335 22447 ts_manager.cc:194] Registered new tserver with Master: afc2290d563b4743b87fac3c7c266de6 (127.0.71.65:42833)
I20260929 01:54:14.559350 22296 ts_manager.cc:194] Registered new tserver with Master: afc2290d563b4743b87fac3c7c266de6 (127.0.71.65:42833)
I20260929 01:54:14.559952 22371 ts_manager.cc:194] Registered new tserver with Master: afc2290d563b4743b87fac3c7c266de6 (127.0.71.65:42833)
I20260929 01:54:14.561549 22371 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:38271
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:54:14.599874 22503 catalog_manager.cc:1308] Loaded cluster ID: 4b631c8567cf41bab15ecc52fd208f7f
I20260929 01:54:14.599988 22503 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73: loading cluster ID for follower catalog manager: success
I20260929 01:54:14.604192 22503 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73: acquiring CA information for follower catalog manager: success
I20260929 01:54:14.606922 22503 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for HTTP/127.0.71.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:54:14.639736 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.66:0
--local_ip_for_outbound_sockets=127.0.71.66
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:54:14.958847 22546 catalog_manager.cc:1308] Loaded cluster ID: 4b631c8567cf41bab15ecc52fd208f7f
I20260929 01:54:14.958997 22546 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: loading cluster ID for follower catalog manager: success
I20260929 01:54:14.963178 22546 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: acquiring CA information for follower catalog manager: success
I20260929 01:54:14.965874 22546 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 01:54:14.991818 22691 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:54:14.992206 22691 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:54:14.992373 22691 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:54:15.001609 22691 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:54:15.001873 22691 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.66
I20260929 01:54:15.017410 22691 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.71.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:54:15.019218 22691 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:54:15.020711 22691 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:54:15.030948 22698 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:15.031054 22697 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:15.032400 22700 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:54:15.033463 22691 server_base.cc:1061] running on GCE node
I20260929 01:54:15.034078 22691 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:54:15.034293 22691 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:54:15.034403 22691 hybrid_clock.cc:648] HybridClock initialized: now 1790646855034374 us; error 0 us; skew 500 ppm
I20260929 01:54:15.034742 22691 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:54:15 dist-test-slave-vz23 krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:54:15.038509 22691 init.cc:377] Logged in from keytab as kudu/127.0.71.66@KRBTEST.COM (short username kudu)
I20260929 01:54:15.040582 22691 webserver.cc:533] Webserver started at http://127.0.71.66:46417/ using document root <none> and password file <none>
I20260929 01:54:15.041364 22691 fs_manager.cc:362] Metadata directory not provided
I20260929 01:54:15.041497 22691 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:54:15.041844 22691 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:54:15.044250 22691 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/data/instance:
uuid: "f95e349e0c094c5ba984109acf0ff50b"
format_stamp: "Formatted at 2026-09-29 01:54:15 on dist-test-slave-vz23"
I20260929 01:54:15.045143 22691 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/wal/instance:
uuid: "f95e349e0c094c5ba984109acf0ff50b"
format_stamp: "Formatted at 2026-09-29 01:54:15 on dist-test-slave-vz23"
I20260929 01:54:15.050653 22691 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:54:15.054222 22706 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:15.055830 22691 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:54:15.056003 22691 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/wal
uuid: "f95e349e0c094c5ba984109acf0ff50b"
format_stamp: "Formatted at 2026-09-29 01:54:15 on dist-test-slave-vz23"
I20260929 01:54:15.056181 22691 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:54:15.094542 22691 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:54:15.099125 22691 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:54:15.099457 22691 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:54:15.100700 22691 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:54:15.102408 22691 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:54:15.102519 22691 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:15.102677 22691 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:54:15.102749 22691 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:15.148487 22691 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.66:39541
I20260929 01:54:15.148550 22818 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.66:39541 every 8 connection(s)
I20260929 01:54:15.150065 22691 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
I20260929 01:54:15.154138 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 22691
I20260929 01:54:15.154358 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/wal/instance
Sep 29 01:54:15 dist-test-slave-vz23 krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:54:15 dist-test-slave-vz23 krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:54:15 dist-test-slave-vz23 krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:54:15.211552 22820 heartbeater.cc:344] Connected to a master server at 127.0.71.126:36771
I20260929 01:54:15.211683 22821 heartbeater.cc:344] Connected to a master server at 127.0.71.125:46761
I20260929 01:54:15.211982 22820 heartbeater.cc:461] Registering TS with master...
I20260929 01:54:15.211980 22821 heartbeater.cc:461] Registering TS with master...
I20260929 01:54:15.213194 22820 heartbeater.cc:507] Master 127.0.71.126:36771 requested a full tablet report, sending...
I20260929 01:54:15.213263 22821 heartbeater.cc:507] Master 127.0.71.125:46761 requested a full tablet report, sending...
I20260929 01:54:15.214962 22371 ts_manager.cc:194] Registered new tserver with Master: f95e349e0c094c5ba984109acf0ff50b (127.0.71.66:39541)
I20260929 01:54:15.215080 22296 ts_manager.cc:194] Registered new tserver with Master: f95e349e0c094c5ba984109acf0ff50b (127.0.71.66:39541)
I20260929 01:54:15.216090 22371 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.66@KRBTEST.COM'} at 127.0.71.66:37373
I20260929 01:54:15.219285 22819 heartbeater.cc:344] Connected to a master server at 127.0.71.124:45059
I20260929 01:54:15.219419 22819 heartbeater.cc:461] Registering TS with master...
I20260929 01:54:15.219744 22819 heartbeater.cc:507] Master 127.0.71.124:45059 requested a full tablet report, sending...
I20260929 01:54:15.221020 22447 ts_manager.cc:194] Registered new tserver with Master: f95e349e0c094c5ba984109acf0ff50b (127.0.71.66:39541)
WARNING: no policy specified for kudu/127.0.71.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:54:15.300886 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.67:0
--local_ip_for_outbound_sockets=127.0.71.67
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:54:15.564904 22679 heartbeater.cc:499] Master 127.0.71.125:46761 was elected leader, sending a full tablet report...
W20260929 01:54:15.673460 22833 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:54:15.673779 22833 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:54:15.673903 22833 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:54:15.682179 22833 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:54:15.682425 22833 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.67
I20260929 01:54:15.693782 22833 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.71.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:54:15.695374 22833 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:54:15.696794 22833 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:54:15.706542 22839 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:15.706519 22838 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:15.707147 22841 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:54:15.707746 22833 server_base.cc:1061] running on GCE node
I20260929 01:54:15.708343 22833 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:54:15.708539 22833 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:54:15.708623 22833 hybrid_clock.cc:648] HybridClock initialized: now 1790646855708603 us; error 0 us; skew 500 ppm
I20260929 01:54:15.708869 22833 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:54:15 dist-test-slave-vz23 krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:54:15.713001 22833 init.cc:377] Logged in from keytab as kudu/127.0.71.67@KRBTEST.COM (short username kudu)
I20260929 01:54:15.714576 22833 webserver.cc:533] Webserver started at http://127.0.71.67:46485/ using document root <none> and password file <none>
I20260929 01:54:15.715273 22833 fs_manager.cc:362] Metadata directory not provided
I20260929 01:54:15.715370 22833 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:54:15.715731 22833 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:54:15.718091 22833 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/data/instance:
uuid: "c70db43b7ed24252857d7e50932909bd"
format_stamp: "Formatted at 2026-09-29 01:54:15 on dist-test-slave-vz23"
I20260929 01:54:15.718816 22833 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/wal/instance:
uuid: "c70db43b7ed24252857d7e50932909bd"
format_stamp: "Formatted at 2026-09-29 01:54:15 on dist-test-slave-vz23"
I20260929 01:54:15.723984 22833 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:54:15.727417 22847 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:15.728879 22833 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:54:15.728999 22833 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/wal
uuid: "c70db43b7ed24252857d7e50932909bd"
format_stamp: "Formatted at 2026-09-29 01:54:15 on dist-test-slave-vz23"
I20260929 01:54:15.729194 22833 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:54:15.748183 22833 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:54:15.752246 22833 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:54:15.752559 22833 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:54:15.753444 22833 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:54:15.754875 22833 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:54:15.754945 22833 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:15.755088 22833 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:54:15.755151 22833 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:15.791946 22833 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.67:32963
I20260929 01:54:15.792035 22959 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.67:32963 every 8 connection(s)
I20260929 01:54:15.793397 22833 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
I20260929 01:54:15.795010 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 22833
I20260929 01:54:15.795219 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/wal/instance
Sep 29 01:54:15 dist-test-slave-vz23 krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:54:15 dist-test-slave-vz23 krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:54:15 dist-test-slave-vz23 krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646855, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:54:15.829584 22961 heartbeater.cc:344] Connected to a master server at 127.0.71.126:36771
I20260929 01:54:15.829991 22961 heartbeater.cc:461] Registering TS with master...
I20260929 01:54:15.830117 22962 heartbeater.cc:344] Connected to a master server at 127.0.71.125:46761
I20260929 01:54:15.830295 22962 heartbeater.cc:461] Registering TS with master...
I20260929 01:54:15.830796 22960 heartbeater.cc:344] Connected to a master server at 127.0.71.124:45059
I20260929 01:54:15.830920 22960 heartbeater.cc:461] Registering TS with master...
I20260929 01:54:15.831300 22962 heartbeater.cc:507] Master 127.0.71.125:46761 requested a full tablet report, sending...
I20260929 01:54:15.831327 22961 heartbeater.cc:507] Master 127.0.71.126:36771 requested a full tablet report, sending...
I20260929 01:54:15.831408 22960 heartbeater.cc:507] Master 127.0.71.124:45059 requested a full tablet report, sending...
I20260929 01:54:15.833050 22371 ts_manager.cc:194] Registered new tserver with Master: c70db43b7ed24252857d7e50932909bd (127.0.71.67:32963)
I20260929 01:54:15.833173 22447 ts_manager.cc:194] Registered new tserver with Master: c70db43b7ed24252857d7e50932909bd (127.0.71.67:32963)
I20260929 01:54:15.833272 22296 ts_manager.cc:194] Registered new tserver with Master: c70db43b7ed24252857d7e50932909bd (127.0.71.67:32963)
I20260929 01:54:15.834640 22371 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.67@KRBTEST.COM'} at 127.0.71.67:47191
I20260929 01:54:15.840620 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:54:15.847496 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:54:15.847654 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 22265
I20260929 01:54:15.848380 22326 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:54:15.958681 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 22265
I20260929 01:54:15.981426 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:36771
--webserver_interface=127.0.71.126
--webserver_port=34037
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:54:16.109833 22358 consensus_peers.cc:597] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 -> Peer 7f6befd994534bbebdb63da555aabb73 (127.0.71.126:36771): Couldn't send request to peer 7f6befd994534bbebdb63da555aabb73. Status: Network error: Client connection negotiation failed: client connection to 127.0.71.126:36771: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260929 01:54:16.217934 22820 heartbeater.cc:646] Failed to heartbeat to 127.0.71.126:36771 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.71.126:36771: connect: Connection refused (error 111)
I20260929 01:54:16.218986 22821 heartbeater.cc:499] Master 127.0.71.125:46761 was elected leader, sending a full tablet report...
W20260929 01:54:16.319682 22971 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:54:16.319985 22971 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:54:16.320084 22971 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:54:16.328547 22971 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:54:16.328657 22971 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:54:16.328778 22971 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:54:16.328840 22971 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:54:16.339998 22971 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:36771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=34037
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:54:16.341707 22971 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:54:16.343037 22971 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:54:16.351151 22980 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:16.351245 22979 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:16.351950 22982 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:54:16.352736 22971 server_base.cc:1061] running on GCE node
I20260929 01:54:16.353447 22971 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:54:16.353587 22971 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:54:16.353654 22971 hybrid_clock.cc:648] HybridClock initialized: now 1790646856353634 us; error 0 us; skew 500 ppm
I20260929 01:54:16.353906 22971 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:54:16 dist-test-slave-vz23 krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:54:16.357383 22971 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:54:16.359263 22971 webserver.cc:533] Webserver started at http://127.0.71.126:34037/ using document root <none> and password file <none>
I20260929 01:54:16.359972 22971 fs_manager.cc:362] Metadata directory not provided
I20260929 01:54:16.360119 22971 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:54:16.365725 22971 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 01:54:16.368490 22988 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:16.369802 22971 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:54:16.369988 22971 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "7f6befd994534bbebdb63da555aabb73"
format_stamp: "Formatted at 2026-09-29 01:54:12 on dist-test-slave-vz23"
I20260929 01:54:16.370692 22971 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:54:16.405282 22971 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:54:16.409128 22971 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:54:16.409399 22971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:54:16.430763 22971 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:36771
I20260929 01:54:16.430784 23039 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:36771 every 8 connection(s)
I20260929 01:54:16.432648 22971 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:54:16.433573 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 22971
I20260929 01:54:16.433931 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 22549
I20260929 01:54:16.440815 23040 sys_catalog.cc:263] Verifying existing consensus state
Sep 29 01:54:16 dist-test-slave-vz23 krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 01:54:16.444655 23040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: Bootstrap starting.
I20260929 01:54:16.451690 22672 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:54:16.464105 23040 log.cc:826] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: Log is configured to *not* fsync() on all Append() calls
I20260929 01:54:16.472683 23040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:54:16.473209 23040 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: Bootstrap complete.
I20260929 01:54:16.480269 23040 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } }
I20260929 01:54:16.481033 23040 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6befd994534bbebdb63da555aabb73, State: Initialized, Role: FOLLOWER
I20260929 01:54:16.481813 23040 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } }
I20260929 01:54:16.482642 23046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:16.482962 23046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:54:16.483711 23040 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:54:16.494316 23057 catalog_manager.cc:1308] Loaded cluster ID: 4b631c8567cf41bab15ecc52fd208f7f
I20260929 01:54:16.494478 23057 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: loading cluster ID for follower catalog manager: success
I20260929 01:54:16.499679 23057 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: acquiring CA information for follower catalog manager: success
I20260929 01:54:16.502977 23057 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:54:16.575749 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 22549
I20260929 01:54:16.594115 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:42833
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--webserver_port=34679
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:54:16.594512 23060 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:54:16.566240 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:37597 (local address 127.0.71.126:36771)
0929 01:54:16.566513 (+ 273us) server_negotiation.cc:207] Beginning negotiation
0929 01:54:16.566524 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0929 01:54:16.575639 (+ 9115us) server_negotiation.cc:408] Connection header received
0929 01:54:16.576023 (+ 384us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:54:16.576030 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:54:16.576103 (+ 73us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:54:16.576189 (+ 86us) server_negotiation.cc:227] Negotiated authn=CERTIFICATE
0929 01:54:16.593186 (+ 16997us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:37597: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":93}
I20260929 01:54:16.605666 22550 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff24fc03c4e74eae9c1472eaa7875378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:16.607067 22550 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [sys.catalog]: This master's current role is: LEADER
I20260929 01:54:16.614625 23046 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff24fc03c4e74eae9c1472eaa7875378. Latest consensus state: current_term: 1 leader_uuid: "ff24fc03c4e74eae9c1472eaa7875378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:16.614998 23046 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:54:16.837567 22962 heartbeater.cc:499] Master 127.0.71.125:46761 was elected leader, sending a full tablet report...
I20260929 01:54:16.845369 23005 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "c70db43b7ed24252857d7e50932909bd" instance_seqno: 1790646855783014) as {username='kudu', principal='kudu/127.0.71.67@KRBTEST.COM'} at 127.0.71.67:57401; Asking this server to re-register.
I20260929 01:54:16.846099 22961 heartbeater.cc:461] Registering TS with master...
I20260929 01:54:16.847645 23005 ts_manager.cc:194] Registered new tserver with Master: c70db43b7ed24252857d7e50932909bd (127.0.71.67:32963)
W20260929 01:54:16.952800 23064 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:54:16.953094 23064 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:54:16.953261 23064 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:54:16.961925 23064 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:54:16.962182 23064 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:54:16.973989 23064 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:42833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=34679
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:54:16.975648 23064 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:54:16.977056 23064 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:54:16.986918 23074 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:16.986894 23073 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:16.987805 23076 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:54:16.988515 23064 server_base.cc:1061] running on GCE node
I20260929 01:54:16.989102 23064 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:54:16.989323 23064 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:54:16.989415 23064 hybrid_clock.cc:648] HybridClock initialized: now 1790646856989379 us; error 0 us; skew 500 ppm
I20260929 01:54:16.989718 23064 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:54:16 dist-test-slave-vz23 krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:54:16.993781 23064 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:54:16.995489 23064 webserver.cc:533] Webserver started at http://127.0.71.65:34679/ using document root <none> and password file <none>
I20260929 01:54:16.996258 23064 fs_manager.cc:362] Metadata directory not provided
I20260929 01:54:16.996414 23064 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:54:17.002640 23064 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 01:54:17.006790 23082 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:17.008785 23064 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.004s
I20260929 01:54:17.008962 23064 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "afc2290d563b4743b87fac3c7c266de6"
format_stamp: "Formatted at 2026-09-29 01:54:14 on dist-test-slave-vz23"
I20260929 01:54:17.009842 23064 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:54:17.038693 23064 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:54:17.042904 23064 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:54:17.043195 23064 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:54:17.044297 23064 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:54:17.045946 23064 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:54:17.046082 23064 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:17.046211 23064 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:54:17.046299 23064 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:17.052135 23065 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "ff24fc03c4e74eae9c1472eaa7875378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:17.052534 23065 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [sys.catalog]: This master's current role is: LEADER
I20260929 01:54:17.081146 23064 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:42833
I20260929 01:54:17.081194 23194 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:42833 every 8 connection(s)
I20260929 01:54:17.082662 23064 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:54:17.083309 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 23064
Sep 29 01:54:17 dist-test-slave-vz23 krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646835, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for hive/127.0.0.1@KRBTEST.COM
Sep 29 01:54:17 dist-test-slave-vz23 krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:54:17 dist-test-slave-vz23 krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:54:17 dist-test-slave-vz23 krb5kdc[21972](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646856, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:54:17.121441 23196 heartbeater.cc:344] Connected to a master server at 127.0.71.126:36771
I20260929 01:54:17.121902 23196 heartbeater.cc:461] Registering TS with master...
I20260929 01:54:17.123013 23196 heartbeater.cc:507] Master 127.0.71.126:36771 requested a full tablet report, sending...
I20260929 01:54:17.125013 23005 ts_manager.cc:194] Registered new tserver with Master: afc2290d563b4743b87fac3c7c266de6 (127.0.71.65:42833)
I20260929 01:54:17.128024 23195 heartbeater.cc:344] Connected to a master server at 127.0.71.124:45059
I20260929 01:54:17.128203 23195 heartbeater.cc:461] Registering TS with master...
I20260929 01:54:17.128872 23195 heartbeater.cc:507] Master 127.0.71.124:45059 requested a full tablet report, sending...
I20260929 01:54:17.129845 23197 heartbeater.cc:344] Connected to a master server at 127.0.71.125:46761
I20260929 01:54:17.130009 23197 heartbeater.cc:461] Registering TS with master...
I20260929 01:54:17.130465 23197 heartbeater.cc:507] Master 127.0.71.125:46761 requested a full tablet report, sending...
I20260929 01:54:17.130585 22447 ts_manager.cc:194] Re-registered known tserver with Master: afc2290d563b4743b87fac3c7c266de6 (127.0.71.65:42833)
I20260929 01:54:17.131914 22371 ts_manager.cc:194] Re-registered known tserver with Master: afc2290d563b4743b87fac3c7c266de6 (127.0.71.65:42833)
I20260929 01:54:17.133308 22371 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:54307
I20260929 01:54:17.227268 22820 heartbeater.cc:344] Connected to a master server at 127.0.71.126:36771
I20260929 01:54:17.228489 23005 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f95e349e0c094c5ba984109acf0ff50b" instance_seqno: 1790646855138044) as {username='kudu', principal='kudu/127.0.71.66@KRBTEST.COM'} at 127.0.71.66:52255; Asking this server to re-register.
I20260929 01:54:17.229123 22820 heartbeater.cc:461] Registering TS with master...
I20260929 01:54:17.230373 23005 ts_manager.cc:194] Registered new tserver with Master: f95e349e0c094c5ba984109acf0ff50b (127.0.71.66:39541)
I20260929 01:54:17.245604 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 23064
W20260929 01:54:17.247046 23210 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:54:17.243053 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:56338 (local address 127.0.71.65:42833)
0929 01:54:17.243942 (+ 889us) server_negotiation.cc:207] Beginning negotiation
0929 01:54:17.243949 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0929 01:54:17.243982 (+ 33us) server_negotiation.cc:408] Connection header received
0929 01:54:17.244047 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:54:17.244054 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:54:17.244262 (+ 208us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:54:17.244425 (+ 163us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:54:17.245572 (+ 1147us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56338: BlockingRecv error: recv got EOF from 127.0.0.1:56338 (error 108)
Metrics: {"server-negotiator.queue_time_us":750,"thread_start_us":523,"threads_started":1}
W20260929 01:54:17.247941 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 23064: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:42833: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:54:17.248093 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 23064
I20260929 01:54:17.264871 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:42833
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--webserver_port=34679
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:54:17.613272 23212 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:54:17.613596 23212 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:54:17.613765 23212 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:54:17.613850 23212 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:54:17.622351 23212 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:54:17.622615 23212 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:54:17.633620 23212 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:42833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=34679
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:54:17.635295 23212 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:54:17.636916 23212 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:54:17.646489 23218 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:17.646633 23217 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:17.647603 23220 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:54:17.647611 23212 server_base.cc:1061] running on GCE node
I20260929 01:54:17.648196 23212 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:54:17.648320 23212 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:54:17.648394 23212 hybrid_clock.cc:648] HybridClock initialized: now 1790646857648366 us; error 0 us; skew 500 ppm
I20260929 01:54:17.648721 23212 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:54:17 dist-test-slave-vz23 krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646857, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:54:17.652118 23212 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:54:17.653892 23212 webserver.cc:533] Webserver started at http://127.0.71.65:34679/ using document root <none> and password file <none>
I20260929 01:54:17.654619 23212 fs_manager.cc:362] Metadata directory not provided
I20260929 01:54:17.654703 23212 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:54:17.660702 23212 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:54:17.664011 23226 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:17.665505 23212 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:54:17.665700 23212 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "afc2290d563b4743b87fac3c7c266de6"
format_stamp: "Formatted at 2026-09-29 01:54:14 on dist-test-slave-vz23"
I20260929 01:54:17.666429 23212 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:54:17.708009 23212 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:54:17.720593 23212 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:54:17.720913 23212 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:54:17.721892 23212 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:54:17.723489 23212 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:54:17.723605 23212 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:17.723757 23212 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:54:17.723829 23212 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:54:17.724231 23212 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:54:17.724350 23212 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:54:17.724442 23212 fault_injection.cc:59] injecting fault for kudu (pid 23212): FLAGS_fault_before_start (process will exit)
I20260929 01:54:17.738579 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:42833
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--webserver_port=34679
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:54:18.086534 23234 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:54:18.086910 23234 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:54:18.087065 23234 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:54:18.095873 23234 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:54:18.096151 23234 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:54:18.107281 23234 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:42833
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=34679
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:54:18.109149 23234 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:54:18.110533 23234 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:54:18.121328 23239 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:18.121345 23240 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:18.121945 23242 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:54:18.122606 23234 server_base.cc:1061] running on GCE node
I20260929 01:54:18.123124 23234 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:54:18.123258 23234 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:54:18.123320 23234 hybrid_clock.cc:648] HybridClock initialized: now 1790646858123303 us; error 0 us; skew 500 ppm
I20260929 01:54:18.123577 23234 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:54:18 dist-test-slave-vz23 krb5kdc[21972](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646858, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:54:18.127097 23234 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:54:18.128875 23234 webserver.cc:533] Webserver started at http://127.0.71.65:34679/ using document root <none> and password file <none>
I20260929 01:54:18.129611 23234 fs_manager.cc:362] Metadata directory not provided
I20260929 01:54:18.129752 23234 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:54:18.135151 23234 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:54:18.138034 23248 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:18.139489 23234 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 01:54:18.139693 23234 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "afc2290d563b4743b87fac3c7c266de6"
format_stamp: "Formatted at 2026-09-29 01:54:14 on dist-test-slave-vz23"
I20260929 01:54:18.140565 23234 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:54:18.169299 23234 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:54:18.173318 23234 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:54:18.173610 23234 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:54:18.174597 23234 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:54:18.176273 23234 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:54:18.176396 23234 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:18.176561 23234 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:54:18.176656 23234 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:18.212231 23234 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:42833
I20260929 01:54:18.212319 23360 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:42833 every 8 connection(s)
I20260929 01:54:18.213831 23234 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:54:18.221773 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 23234
I20260929 01:54:18.222174 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 23234
W20260929 01:54:18.242619 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 23234: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:42833: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:54:18.242830 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 23234
I20260929 01:54:18.260030 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 22691
W20260929 01:54:18.260197 23365 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:54:18.217294 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:33923 (local address 127.0.71.126:36771)
0929 01:54:18.217730 (+ 436us) server_negotiation.cc:207] Beginning negotiation
0929 01:54:18.217739 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0929 01:54:18.219379 (+ 1640us) server_negotiation.cc:408] Connection header received
0929 01:54:18.232715 (+ 13336us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:54:18.232722 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:54:18.232806 (+ 84us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:54:18.232900 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:54:18.240638 (+ 7738us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:54:18.241665 (+ 1027us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:54:18.259854 (+ 18189us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:33923: BlockingRecv error: recv got EOF from 127.0.71.65:33923 (error 108)
Metrics: {"server-negotiator.queue_time_us":317,"thread_start_us":435,"threads_started":1}
W20260929 01:54:18.260849 23370 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:54:18.220484 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:51897 (local address 127.0.71.124:45059)
0929 01:54:18.240490 (+ 20006us) server_negotiation.cc:207] Beginning negotiation
0929 01:54:18.240498 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0929 01:54:18.259769 (+ 19271us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:51897: BlockingRecv error: recv got EOF from 127.0.71.65:51897 (error 108)
Metrics: {"server-negotiator.queue_time_us":19874,"thread_start_us":18518,"threads_started":1}
W20260929 01:54:18.260907 23366 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:54:18.217586 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:58589 (local address 127.0.71.125:46761)
0929 01:54:18.236568 (+ 18982us) server_negotiation.cc:207] Beginning negotiation
0929 01:54:18.236575 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0929 01:54:18.236597 (+ 22us) server_negotiation.cc:408] Connection header received
0929 01:54:18.236674 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:54:18.236682 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:54:18.236821 (+ 139us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:54:18.236965 (+ 144us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:54:18.238900 (+ 1935us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:54:18.240476 (+ 1576us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:54:18.244256 (+ 3780us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:54:18.244845 (+ 589us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0929 01:54:18.259882 (+ 15037us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:58589: BlockingRecv error: recv got EOF from 127.0.71.65:58589 (error 108)
Metrics: {"server-negotiator.queue_time_us":18865,"thread_start_us":19471,"threads_started":1}
W20260929 01:54:18.262919 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 22691: Not authorized: Client connection negotiation failed: client connection to 127.0.71.66:39541: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:54:18.263104 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 22691
I20260929 01:54:18.279505 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 22833
W20260929 01:54:18.282137 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 22833: Not authorized: Client connection negotiation failed: client connection to 127.0.71.67:32963: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:54:18.282290 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 22833
I20260929 01:54:18.298715 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 22971
W20260929 01:54:18.301067 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 22971: Not authorized: Client connection negotiation failed: client connection to 127.0.71.126:36771: server requires authentication, but client does not have Kerberos credentials available
W20260929 01:54:18.301167 23365 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:54:18.299762 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:38792 (local address 127.0.71.126:36771)
0929 01:54:18.299964 (+ 202us) server_negotiation.cc:207] Beginning negotiation
0929 01:54:18.299969 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0929 01:54:18.299989 (+ 20us) server_negotiation.cc:408] Connection header received
0929 01:54:18.300052 (+ 63us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:54:18.300057 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:54:18.300123 (+ 66us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:54:18.300220 (+ 97us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:54:18.300973 (+ 753us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:38792: BlockingRecv error: recv got EOF from 127.0.0.1:38792 (error 108)
Metrics: {"server-negotiator.queue_time_us":83}
I20260929 01:54:18.301263 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 22971
I20260929 01:54:18.320983 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 22340
I20260929 01:54:18.322175 22401 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:54:18.437109 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 22340
I20260929 01:54:18.458259 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 22416
I20260929 01:54:18.459467 22477 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:54:18.569355 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 22416
Shutting down hive metastore.
Sep 29 01:54:18 dist-test-slave-vz23 krb5kdc[23383](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:54:18 dist-test-slave-vz23 krb5kdc[23383](info): set up 2 sockets
Sep 29 01:54:18 dist-test-slave-vz23 krb5kdc[23383](info): commencing operation
krb5kdc: starting...
W20260929 01:54:20.983526 285 mini_kdc.cc:121] Time spent starting KDC: real 2.018s user 0.004s sys 0.014s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-admin@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-user@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "joe-interloper@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 01:54:21 dist-test-slave-vz23 krb5kdc[23383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for hive/127.0.0.1@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "hive/127.0.0.1@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal hive/127.0.0.1 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/hive_127.0.0.1.keytab.
Entry for principal hive/127.0.0.1 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/hive_127.0.0.1.keytab.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5.conf
2026-09-29 01:54:22: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
Sep 29 01:54:24 dist-test-slave-vz23 krb5kdc[23383](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790646864, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 01:54:28 dist-test-slave-vz23 krb5kdc[23383](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790646868, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 01:54:28 dist-test-slave-vz23 krb5kdc[23383](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790646868, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.126@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.126@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:54:30.660506 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:36771
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:54:30.992337 23574 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:54:30.992698 23574 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:54:30.992790 23574 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:54:31.001165 23574 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:54:31.001277 23574 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:54:31.001334 23574 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:54:31.001384 23574 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:54:31.018245 23574 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:36771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:54:31.019816 23574 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:54:31.021155 23574 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:54:31.029528 23579 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:31.029662 23580 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:31.030465 23582 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:54:31.030834 23574 server_base.cc:1061] running on GCE node
I20260929 01:54:31.031590 23574 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:54:31.031769 23574 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:54:31.031876 23574 hybrid_clock.cc:648] HybridClock initialized: now 1790646871031837 us; error 0 us; skew 500 ppm
I20260929 01:54:31.032238 23574 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:54:31 dist-test-slave-vz23 krb5kdc[23383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:54:31.036013 23574 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:54:31.038012 23574 webserver.cc:533] Webserver started at http://127.0.71.126:46451/ using document root <none> and password file <none>
I20260929 01:54:31.038753 23574 fs_manager.cc:362] Metadata directory not provided
I20260929 01:54:31.038893 23574 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:54:31.044548 23574 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:54:31.047544 23588 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:31.049072 23574 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:54:31.049248 23574 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "7f6befd994534bbebdb63da555aabb73"
format_stamp: "Formatted at 2026-09-29 01:54:12 on dist-test-slave-vz23"
I20260929 01:54:31.050012 23574 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:54:31.072074 23574 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:54:31.084942 23574 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:54:31.085256 23574 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:54:31.106482 23574 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:36771
I20260929 01:54:31.106528 23639 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:36771 every 8 connection(s)
I20260929 01:54:31.108201 23574 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:54:31.113976 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 23574
I20260929 01:54:31.114228 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:54:31.115763 23640 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:54:31.119401 23640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: Bootstrap starting.
WARNING: no policy specified for kudu/127.0.71.125@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.125@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:54:31.137029 23640 log.cc:826] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: Log is configured to *not* fsync() on all Append() calls
I20260929 01:54:31.147886 23640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:54:31.148584 23640 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: Bootstrap complete.
I20260929 01:54:31.159657 23640 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } }
I20260929 01:54:31.160377 23640 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6befd994534bbebdb63da555aabb73, State: Initialized, Role: FOLLOWER
I20260929 01:54:31.161269 23640 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } }
I20260929 01:54:31.162127 23645 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:31.162451 23645 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:54:31.163228 23640 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [sys.catalog]: configured and running, proceeding with master startup.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.125 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.125 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:54:31.173940 23656 catalog_manager.cc:1308] Loaded cluster ID: 4b631c8567cf41bab15ecc52fd208f7f
I20260929 01:54:31.174152 23656 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: loading cluster ID for follower catalog manager: success
I20260929 01:54:31.178735 23656 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: acquiring CA information for follower catalog manager: success
I20260929 01:54:31.181558 23656 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for HTTP/127.0.71.125@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.125@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:54:31.197932 23574 master_runner.cc:187] Error getting master registration for 127.0.71.125:46761: Network error: Client connection negotiation failed: client connection to 127.0.71.125:46761: connect: Connection refused (error 111),
I20260929 01:54:31.201758 23574 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.125 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.125 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:54:31.228797 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:46761
--webserver_interface=127.0.71.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:54:31.577957 23662 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:54:31.578346 23662 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:54:31.578506 23662 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:54:31.587121 23662 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:54:31.587291 23662 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:54:31.587405 23662 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:54:31.587486 23662 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:54:31.598560 23662 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:46761
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:54:31.600214 23662 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:54:31.601641 23662 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:54:31.610383 23667 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:31.610549 23668 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:31.610788 23670 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:54:31.611661 23662 server_base.cc:1061] running on GCE node
I20260929 01:54:31.612401 23662 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:54:31.612623 23662 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:54:31.612694 23662 hybrid_clock.cc:648] HybridClock initialized: now 1790646871612677 us; error 0 us; skew 500 ppm
I20260929 01:54:31.612974 23662 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:54:31 dist-test-slave-vz23 krb5kdc[23383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:54:31.616741 23662 init.cc:377] Logged in from keytab as kudu/127.0.71.125@KRBTEST.COM (short username kudu)
I20260929 01:54:31.618913 23662 webserver.cc:533] Webserver started at http://127.0.71.125:38753/ using document root <none> and password file <none>
I20260929 01:54:31.619635 23662 fs_manager.cc:362] Metadata directory not provided
I20260929 01:54:31.619776 23662 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:54:31.625514 23662 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:54:31.628736 23676 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:31.630173 23662 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:54:31.630353 23662 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "ff24fc03c4e74eae9c1472eaa7875378"
format_stamp: "Formatted at 2026-09-29 01:54:12 on dist-test-slave-vz23"
I20260929 01:54:31.631072 23662 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:54:31.650420 23662 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:54:31.654680 23662 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:54:31.655045 23662 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:54:31.675976 23662 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:46761
I20260929 01:54:31.675997 23727 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:46761 every 8 connection(s)
I20260929 01:54:31.677708 23662 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:54:31.685333 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 23662
I20260929 01:54:31.685518 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:54:31.686074 23728 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:54:31.689669 23728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378: Bootstrap starting.
WARNING: no policy specified for kudu/127.0.71.124@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.124@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:54:31.711423 23728 log.cc:826] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378: Log is configured to *not* fsync() on all Append() calls
I20260929 01:54:31.721186 23728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:54:31.721838 23728 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378: Bootstrap complete.
I20260929 01:54:31.733593 23728 raft_consensus.cc:359] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } }
I20260929 01:54:31.734719 23728 raft_consensus.cc:740] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: ff24fc03c4e74eae9c1472eaa7875378, State: Initialized, Role: FOLLOWER
I20260929 01:54:31.735633 23728 consensus_queue.cc:260] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } }
I20260929 01:54:31.736752 23733 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:31.737023 23733 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:54:31.738065 23728 sys_catalog.cc:565] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [sys.catalog]: configured and running, proceeding with master startup.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.124 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.124 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:54:31.757663 23744 catalog_manager.cc:1308] Loaded cluster ID: 4b631c8567cf41bab15ecc52fd208f7f
I20260929 01:54:31.757877 23744 catalog_manager.cc:1601] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378: loading cluster ID for follower catalog manager: success
I20260929 01:54:31.762264 23744 catalog_manager.cc:1623] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378: acquiring CA information for follower catalog manager: success
Sep 29 01:54:31 dist-test-slave-vz23 krb5kdc[23383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646871, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:54:31.765733 23744 catalog_manager.cc:1651] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for HTTP/127.0.71.124@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.124@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:54:31.794044 23662 master_runner.cc:187] Error getting master registration for 127.0.71.124:45059: Network error: Client connection negotiation failed: client connection to 127.0.71.124:45059: connect: Connection refused (error 111),
I20260929 01:54:31.799551 23662 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.124 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.124 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:54:31.808332 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:45059
--webserver_interface=127.0.71.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:54:32.154279 23751 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:54:32.154624 23751 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:54:32.154721 23751 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:54:32.163494 23751 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:54:32.163609 23751 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:54:32.163664 23751 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:54:32.163704 23751 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:54:32.174820 23751 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:45059
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:54:32.176358 23751 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:54:32.177711 23751 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:54:32.186283 23757 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:32.186975 23756 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:32.187368 23759 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:54:32.187716 23751 server_base.cc:1061] running on GCE node
I20260929 01:54:32.188548 23751 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:54:32.188721 23751 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:54:32.188787 23751 hybrid_clock.cc:648] HybridClock initialized: now 1790646872188770 us; error 0 us; skew 500 ppm
I20260929 01:54:32.189100 23751 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:54:32 dist-test-slave-vz23 krb5kdc[23383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:54:32.193156 23751 init.cc:377] Logged in from keytab as kudu/127.0.71.124@KRBTEST.COM (short username kudu)
I20260929 01:54:32.195120 23751 webserver.cc:533] Webserver started at http://127.0.71.124:43037/ using document root <none> and password file <none>
I20260929 01:54:32.195858 23751 fs_manager.cc:362] Metadata directory not provided
I20260929 01:54:32.195971 23751 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:54:32.201834 23751 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:54:32.204977 23765 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:32.206449 23751 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:54:32.206589 23751 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "84706cb3b1a64ebc9710b0c330b6ea73"
format_stamp: "Formatted at 2026-09-29 01:54:13 on dist-test-slave-vz23"
I20260929 01:54:32.207274 23751 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:54:32.235483 23751 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:54:32.239739 23751 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:54:32.239969 23751 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:54:32.261166 23751 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:45059
I20260929 01:54:32.261201 23816 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:45059 every 8 connection(s)
I20260929 01:54:32.262827 23751 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:54:32.270308 23817 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:54:32.272372 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 23751
I20260929 01:54:32.272611 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:54:32.274284 23817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73: Bootstrap starting.
Sep 29 01:54:32 dist-test-slave-vz23 krb5kdc[23383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:54:32.296530 23817 log.cc:826] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73: Log is configured to *not* fsync() on all Append() calls
Sep 29 01:54:32 dist-test-slave-vz23 krb5kdc[23383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:54:32.306994 23817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:54:32.307677 23817 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73: Bootstrap complete.
I20260929 01:54:32.315141 23817 raft_consensus.cc:359] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } }
I20260929 01:54:32.315881 23817 raft_consensus.cc:740] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 84706cb3b1a64ebc9710b0c330b6ea73, State: Initialized, Role: FOLLOWER
I20260929 01:54:32.316751 23817 consensus_queue.cc:260] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } }
I20260929 01:54:32.317669 23822 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:32.317953 23822 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:54:32.318818 23817 sys_catalog.cc:565] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 01:54:32 dist-test-slave-vz23 krb5kdc[23383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:54:32.335105 23833 catalog_manager.cc:1308] Loaded cluster ID: 4b631c8567cf41bab15ecc52fd208f7f
I20260929 01:54:32.335223 23833 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73: loading cluster ID for follower catalog manager: success
I20260929 01:54:32.339787 23833 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73: acquiring CA information for follower catalog manager: success
I20260929 01:54:32.344149 23833 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Sep 29 01:54:32 dist-test-slave-vz23 krb5kdc[23383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646872, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:54:32.373060 23751 master_runner.cc:264] No leader master found from master 84706cb3b1a64ebc9710b0c330b6ea73
I20260929 01:54:32.373217 23751 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:54:32.825191 23836 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:54:32.825456 23836 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } }
I20260929 01:54:32.827427 23836 leader_election.cc:290] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers ff24fc03c4e74eae9c1472eaa7875378 (127.0.71.125:46761), 84706cb3b1a64ebc9710b0c330b6ea73 (127.0.71.124:45059)
I20260929 01:54:32.835659 23792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f6befd994534bbebdb63da555aabb73" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" is_pre_election: true
I20260929 01:54:32.835700 23703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f6befd994534bbebdb63da555aabb73" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff24fc03c4e74eae9c1472eaa7875378" is_pre_election: true
I20260929 01:54:32.836179 23792 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f6befd994534bbebdb63da555aabb73 in term 1.
I20260929 01:54:32.836205 23703 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7f6befd994534bbebdb63da555aabb73 in term 1.
I20260929 01:54:32.836838 23591 leader_election.cc:304] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f6befd994534bbebdb63da555aabb73, ff24fc03c4e74eae9c1472eaa7875378; no voters:
I20260929 01:54:32.837263 23836 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 01:54:32.837414 23836 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:54:32.837520 23836 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:54:32.842109 23836 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } }
I20260929 01:54:32.842921 23836 leader_election.cc:290] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [CANDIDATE]: Term 2 election: Requested vote from peers ff24fc03c4e74eae9c1472eaa7875378 (127.0.71.125:46761), 84706cb3b1a64ebc9710b0c330b6ea73 (127.0.71.124:45059)
I20260929 01:54:32.843453 23703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f6befd994534bbebdb63da555aabb73" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "ff24fc03c4e74eae9c1472eaa7875378"
I20260929 01:54:32.843508 23792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7f6befd994534bbebdb63da555aabb73" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "84706cb3b1a64ebc9710b0c330b6ea73"
I20260929 01:54:32.843664 23703 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:54:32.843713 23792 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:54:32.847302 23703 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f6befd994534bbebdb63da555aabb73 in term 2.
I20260929 01:54:32.847316 23792 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7f6befd994534bbebdb63da555aabb73 in term 2.
I20260929 01:54:32.847815 23591 leader_election.cc:304] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f6befd994534bbebdb63da555aabb73, ff24fc03c4e74eae9c1472eaa7875378; no voters:
I20260929 01:54:32.848170 23836 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:54:32.848767 23836 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [term 2 LEADER]: Becoming Leader. State: Replica: 7f6befd994534bbebdb63da555aabb73, State: Running, Role: LEADER
I20260929 01:54:32.849259 23836 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } }
I20260929 01:54:32.851742 23839 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f6befd994534bbebdb63da555aabb73. Latest consensus state: current_term: 2 leader_uuid: "7f6befd994534bbebdb63da555aabb73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:32.852072 23839 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [sys.catalog]: This master's current role is: LEADER
I20260929 01:54:32.852790 23841 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:54:32.854651 23841 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:54:32.855577 23841 catalog_manager.cc:1308] Loaded cluster ID: 4b631c8567cf41bab15ecc52fd208f7f
I20260929 01:54:32.855647 23841 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:54:32.856853 23841 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:54:32.857662 23841 catalog_manager.cc:6230] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: Loaded TSK: 0
I20260929 01:54:32.858560 23841 catalog_manager.cc:1563] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.65@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.65@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:54:32.972218 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:54:33.315970 23846 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:54:33.316284 23846 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:54:33.316376 23846 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:54:33.325340 23846 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:54:33.325539 23846 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:54:33.329763 23792 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [term 2 FOLLOWER]: Refusing update from remote peer 7f6befd994534bbebdb63da555aabb73: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 01:54:33.330852 23839 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:54:33.334915 23850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f6befd994534bbebdb63da555aabb73. Latest consensus state: current_term: 2 leader_uuid: "7f6befd994534bbebdb63da555aabb73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:33.335295 23850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:54:33.336948 23836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "7f6befd994534bbebdb63da555aabb73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:33.337167 23836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [sys.catalog]: This master's current role is: LEADER
I20260929 01:54:33.338176 23703 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [term 2 FOLLOWER]: Refusing update from remote peer 7f6befd994534bbebdb63da555aabb73: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 01:54:33.339133 23836 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [LEADER]: Connected to new peer: Peer: permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:54:33.339978 23846 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:54:33.342633 23846 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:54:33.344949 23846 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260929 01:54:33.347728 23850 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "7f6befd994534bbebdb63da555aabb73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:33.347960 23850 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:54:33.351900 23858 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "7f6befd994534bbebdb63da555aabb73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:33.352262 23858 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:54:33.352643 23855 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7f6befd994534bbebdb63da555aabb73. Latest consensus state: current_term: 2 leader_uuid: "7f6befd994534bbebdb63da555aabb73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:33.352880 23855 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:54:33.357647 23860 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:33.357674 23861 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:33.358014 23863 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:54:33.359400 23846 server_base.cc:1061] running on GCE node
I20260929 01:54:33.360013 23846 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:54:33.360185 23846 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:54:33.360304 23846 hybrid_clock.cc:648] HybridClock initialized: now 1790646873360261 us; error 0 us; skew 500 ppm
I20260929 01:54:33.360679 23846 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:54:33 dist-test-slave-vz23 krb5kdc[23383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:54:33.364688 23846 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:54:33.366564 23846 webserver.cc:533] Webserver started at http://127.0.71.65:44615/ using document root <none> and password file <none>
I20260929 01:54:33.367318 23846 fs_manager.cc:362] Metadata directory not provided
I20260929 01:54:33.367441 23846 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:54:33.373054 23846 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:54:33.375967 23869 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:33.377559 23846 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:54:33.377769 23846 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "afc2290d563b4743b87fac3c7c266de6"
format_stamp: "Formatted at 2026-09-29 01:54:14 on dist-test-slave-vz23"
I20260929 01:54:33.378501 23846 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:54:33.411036 23846 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:54:33.415274 23846 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:54:33.415558 23846 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:54:33.416622 23846 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:54:33.418218 23846 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:54:33.418342 23846 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:33.418473 23846 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:54:33.418560 23846 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:33.454226 23846 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:33083
I20260929 01:54:33.454315 23981 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:33083 every 8 connection(s)
I20260929 01:54:33.455780 23846 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:54:33.456976 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 23846
I20260929 01:54:33.457170 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
Sep 29 01:54:33 dist-test-slave-vz23 krb5kdc[23383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:54:33.492853 23982 heartbeater.cc:344] Connected to a master server at 127.0.71.124:45059
I20260929 01:54:33.493316 23982 heartbeater.cc:461] Registering TS with master...
I20260929 01:54:33.494475 23982 heartbeater.cc:507] Master 127.0.71.124:45059 requested a full tablet report, sending...
I20260929 01:54:33.497345 23782 ts_manager.cc:194] Registered new tserver with Master: afc2290d563b4743b87fac3c7c266de6 (127.0.71.65:33083)
Sep 29 01:54:33 dist-test-slave-vz23 krb5kdc[23383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.66@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.66@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Sep 29 01:54:33 dist-test-slave-vz23 krb5kdc[23383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:54:33.506503 23984 heartbeater.cc:344] Connected to a master server at 127.0.71.125:46761
I20260929 01:54:33.506695 23984 heartbeater.cc:461] Registering TS with master...
I20260929 01:54:33.507186 23984 heartbeater.cc:507] Master 127.0.71.125:46761 requested a full tablet report, sending...
I20260929 01:54:33.509142 23693 ts_manager.cc:194] Registered new tserver with Master: afc2290d563b4743b87fac3c7c266de6 (127.0.71.65:33083)
I20260929 01:54:33.511472 23983 heartbeater.cc:344] Connected to a master server at 127.0.71.126:36771
I20260929 01:54:33.511641 23983 heartbeater.cc:461] Registering TS with master...
I20260929 01:54:33.512161 23983 heartbeater.cc:507] Master 127.0.71.126:36771 requested a full tablet report, sending...
I20260929 01:54:33.513842 23605 ts_manager.cc:194] Registered new tserver with Master: afc2290d563b4743b87fac3c7c266de6 (127.0.71.65:33083)
I20260929 01:54:33.515502 23605 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:45259
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.66 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.66 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.66@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.66@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.66 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.66 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:54:33.584775 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.66:0
--local_ip_for_outbound_sockets=127.0.71.66
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:54:33.926282 23996 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:54:33.926630 23996 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:54:33.926803 23996 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:54:33.935216 23996 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:54:33.935477 23996 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.66
I20260929 01:54:33.941650 23836 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "7f6befd994534bbebdb63da555aabb73" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:33.942030 23836 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [sys.catalog]: This master's current role is: LEADER
I20260929 01:54:33.947063 23996 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.71.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:54:33.948658 23996 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:54:33.950004 23996 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:54:33.960696 24005 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:33.960697 24004 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:33.960865 24007 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:54:33.961642 23996 server_base.cc:1061] running on GCE node
I20260929 01:54:33.962271 23996 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:54:33.962446 23996 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:54:33.962558 23996 hybrid_clock.cc:648] HybridClock initialized: now 1790646873962535 us; error 0 us; skew 500 ppm
I20260929 01:54:33.962877 23996 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:54:33 dist-test-slave-vz23 krb5kdc[23383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:54:33.967146 23996 init.cc:377] Logged in from keytab as kudu/127.0.71.66@KRBTEST.COM (short username kudu)
I20260929 01:54:33.969120 23996 webserver.cc:533] Webserver started at http://127.0.71.66:39241/ using document root <none> and password file <none>
I20260929 01:54:33.969897 23996 fs_manager.cc:362] Metadata directory not provided
I20260929 01:54:33.970053 23996 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:54:33.975973 23996 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:54:33.979292 24013 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:33.980878 23996 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 01:54:33.981060 23996 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/wal
uuid: "f95e349e0c094c5ba984109acf0ff50b"
format_stamp: "Formatted at 2026-09-29 01:54:15 on dist-test-slave-vz23"
I20260929 01:54:33.981830 23996 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:54:34.008044 23996 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:54:34.012545 23996 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:54:34.012871 23996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:54:34.013913 23996 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:54:34.015439 23996 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:54:34.015561 23996 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:34.015687 23996 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:54:34.015784 23996 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:34.050339 23996 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.66:35595
I20260929 01:54:34.050396 24125 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.66:35595 every 8 connection(s)
I20260929 01:54:34.051903 23996 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
I20260929 01:54:34.058723 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 23996
I20260929 01:54:34.058943 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-1/wal/instance
Sep 29 01:54:34 dist-test-slave-vz23 krb5kdc[23383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:54:34 dist-test-slave-vz23 krb5kdc[23383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:54:34 dist-test-slave-vz23 krb5kdc[23383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646873, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:54:34.084107 24126 heartbeater.cc:344] Connected to a master server at 127.0.71.124:45059
I20260929 01:54:34.084316 24128 heartbeater.cc:344] Connected to a master server at 127.0.71.125:46761
I20260929 01:54:34.084666 24126 heartbeater.cc:461] Registering TS with master...
I20260929 01:54:34.084672 24128 heartbeater.cc:461] Registering TS with master...
I20260929 01:54:34.085999 24126 heartbeater.cc:507] Master 127.0.71.124:45059 requested a full tablet report, sending...
I20260929 01:54:34.086023 24128 heartbeater.cc:507] Master 127.0.71.125:46761 requested a full tablet report, sending...
I20260929 01:54:34.088099 23693 ts_manager.cc:194] Registered new tserver with Master: f95e349e0c094c5ba984109acf0ff50b (127.0.71.66:35595)
I20260929 01:54:34.088408 23782 ts_manager.cc:194] Registered new tserver with Master: f95e349e0c094c5ba984109acf0ff50b (127.0.71.66:35595)
I20260929 01:54:34.088862 24127 heartbeater.cc:344] Connected to a master server at 127.0.71.126:36771
I20260929 01:54:34.089177 24127 heartbeater.cc:461] Registering TS with master...
I20260929 01:54:34.089712 24127 heartbeater.cc:507] Master 127.0.71.126:36771 requested a full tablet report, sending...
I20260929 01:54:34.090842 23605 ts_manager.cc:194] Registered new tserver with Master: f95e349e0c094c5ba984109acf0ff50b (127.0.71.66:35595)
WARNING: no policy specified for kudu/127.0.71.67@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.67@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:54:34.092149 23605 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.66@KRBTEST.COM'} at 127.0.71.66:34275
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.67 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.67 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.67@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.67@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.67 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.67 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:54:34.172240 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.67:0
--local_ip_for_outbound_sockets=127.0.71.67
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:54:34.518275 23983 heartbeater.cc:499] Master 127.0.71.126:36771 was elected leader, sending a full tablet report...
W20260929 01:54:34.521268 24140 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:54:34.521575 24140 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:54:34.521759 24140 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:54:34.530398 24140 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:54:34.530715 24140 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.67
I20260929 01:54:34.542006 24140 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.71.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:54:34.543628 24140 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:54:34.544981 24140 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:54:34.555179 24145 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:34.555604 24148 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:34.555109 24146 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:54:34.556808 24140 server_base.cc:1061] running on GCE node
I20260929 01:54:34.557463 24140 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:54:34.557623 24140 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:54:34.557701 24140 hybrid_clock.cc:648] HybridClock initialized: now 1790646874557678 us; error 0 us; skew 500 ppm
I20260929 01:54:34.557976 24140 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:54:34 dist-test-slave-vz23 krb5kdc[23383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:54:34.562009 24140 init.cc:377] Logged in from keytab as kudu/127.0.71.67@KRBTEST.COM (short username kudu)
I20260929 01:54:34.563544 24140 webserver.cc:533] Webserver started at http://127.0.71.67:42559/ using document root <none> and password file <none>
I20260929 01:54:34.564237 24140 fs_manager.cc:362] Metadata directory not provided
I20260929 01:54:34.564338 24140 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:54:34.569939 24140 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.001s sys 0.003s
I20260929 01:54:34.572817 24154 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:34.574201 24140 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:54:34.574404 24140 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/wal
uuid: "c70db43b7ed24252857d7e50932909bd"
format_stamp: "Formatted at 2026-09-29 01:54:15 on dist-test-slave-vz23"
I20260929 01:54:34.575212 24140 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:54:34.611415 24140 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:54:34.615401 24140 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:54:34.615671 24140 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:54:34.616710 24140 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:54:34.618242 24140 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:54:34.618359 24140 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:34.618481 24140 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:54:34.618566 24140 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:34.653517 24140 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.67:38217
I20260929 01:54:34.653584 24266 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.67:38217 every 8 connection(s)
I20260929 01:54:34.655288 24140 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
I20260929 01:54:34.656330 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 24140
I20260929 01:54:34.656556 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-2/wal/instance
Sep 29 01:54:34 dist-test-slave-vz23 krb5kdc[23383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:54:34 dist-test-slave-vz23 krb5kdc[23383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:54:34 dist-test-slave-vz23 krb5kdc[23383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646874, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:54:34.686771 24269 heartbeater.cc:344] Connected to a master server at 127.0.71.125:46761
I20260929 01:54:34.686866 24268 heartbeater.cc:344] Connected to a master server at 127.0.71.126:36771
I20260929 01:54:34.687191 24268 heartbeater.cc:461] Registering TS with master...
I20260929 01:54:34.687193 24269 heartbeater.cc:461] Registering TS with master...
I20260929 01:54:34.688513 24267 heartbeater.cc:344] Connected to a master server at 127.0.71.124:45059
I20260929 01:54:34.688609 24268 heartbeater.cc:507] Master 127.0.71.126:36771 requested a full tablet report, sending...
I20260929 01:54:34.688686 24267 heartbeater.cc:461] Registering TS with master...
I20260929 01:54:34.688695 24269 heartbeater.cc:507] Master 127.0.71.125:46761 requested a full tablet report, sending...
I20260929 01:54:34.689205 24267 heartbeater.cc:507] Master 127.0.71.124:45059 requested a full tablet report, sending...
I20260929 01:54:34.690491 23605 ts_manager.cc:194] Registered new tserver with Master: c70db43b7ed24252857d7e50932909bd (127.0.71.67:38217)
I20260929 01:54:34.690568 23782 ts_manager.cc:194] Registered new tserver with Master: c70db43b7ed24252857d7e50932909bd (127.0.71.67:38217)
I20260929 01:54:34.690713 23692 ts_manager.cc:194] Registered new tserver with Master: c70db43b7ed24252857d7e50932909bd (127.0.71.67:38217)
I20260929 01:54:34.691490 23605 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.67@KRBTEST.COM'} at 127.0.71.67:45617
I20260929 01:54:34.696156 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:54:34.703845 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:54:34.704053 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 23574
I20260929 01:54:34.704972 23635 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:54:34.817385 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 23574
W20260929 01:54:34.838140 23766 connection.cc:573] server connection from 127.0.0.1:52698 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:54:34.839210 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:36771
--webserver_interface=127.0.71.126
--webserver_port=46451
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:54:35.098910 24127 heartbeater.cc:499] Master 127.0.71.126:36771 was elected leader, sending a full tablet report...
W20260929 01:54:35.100634 24127 heartbeater.cc:646] Failed to heartbeat to 127.0.71.126:36771 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.71.126:36771: connect: Connection refused (error 111)
W20260929 01:54:35.186780 24278 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:54:35.187160 24278 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:54:35.187328 24278 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:54:35.195953 24278 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:54:35.196067 24278 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:54:35.196170 24278 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:54:35.196236 24278 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:54:35.207420 24278 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:36771
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=46451
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:54:35.209223 24278 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:54:35.210593 24278 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:54:35.219017 24284 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:35.219226 24287 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:35.219450 24285 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:54:35.220139 24278 server_base.cc:1061] running on GCE node
I20260929 01:54:35.220974 24278 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:54:35.221128 24278 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:54:35.221195 24278 hybrid_clock.cc:648] HybridClock initialized: now 1790646875221178 us; error 0 us; skew 500 ppm
I20260929 01:54:35.221468 24278 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:54:35 dist-test-slave-vz23 krb5kdc[23383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:54:35.225502 24278 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:54:35.227463 24278 webserver.cc:533] Webserver started at http://127.0.71.126:46451/ using document root <none> and password file <none>
I20260929 01:54:35.228163 24278 fs_manager.cc:362] Metadata directory not provided
I20260929 01:54:35.228259 24278 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:54:35.234714 24278 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:54:35.237787 24293 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:35.239372 24278 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:54:35.239542 24278 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "7f6befd994534bbebdb63da555aabb73"
format_stamp: "Formatted at 2026-09-29 01:54:12 on dist-test-slave-vz23"
I20260929 01:54:35.240542 24278 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:54:35.264199 24278 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:54:35.272197 24278 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:54:35.272676 24278 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:54:35.296649 24344 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:36771 every 8 connection(s)
I20260929 01:54:35.296679 24278 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:36771
I20260929 01:54:35.298437 24278 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:54:35.303798 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 24278
I20260929 01:54:35.304190 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 23846
I20260929 01:54:35.306125 24345 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:54:35.310232 24345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: Bootstrap starting.
Sep 29 01:54:35 dist-test-slave-vz23 krb5kdc[23383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 01:54:35.328969 24345 log.cc:826] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: Log is configured to *not* fsync() on all Append() calls
I20260929 01:54:35.332131 23977 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:54:35.340765 24345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:54:35.341409 24345 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: Bootstrap complete.
I20260929 01:54:35.348591 24345 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } }
I20260929 01:54:35.349328 24345 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7f6befd994534bbebdb63da555aabb73, State: Initialized, Role: FOLLOWER
I20260929 01:54:35.350132 24345 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } }
I20260929 01:54:35.351011 24351 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:35.351280 24351 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:54:35.352131 24345 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:54:35.380152 24362 catalog_manager.cc:1308] Loaded cluster ID: 4b631c8567cf41bab15ecc52fd208f7f
I20260929 01:54:35.380276 24362 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: loading cluster ID for follower catalog manager: success
Sep 29 01:54:35 dist-test-slave-vz23 krb5kdc[23383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:54:35.384608 24362 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: acquiring CA information for follower catalog manager: success
I20260929 01:54:35.387475 24362 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:54:35.412197 24278 master_runner.cc:264] No leader master found from master 7f6befd994534bbebdb63da555aabb73
I20260929 01:54:35.412353 24278 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:54:35.472491 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 23846
I20260929 01:54:35.489550 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:33083
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--webserver_port=44615
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:54:35.694380 24268 heartbeater.cc:499] Master 127.0.71.126:36771 was elected leader, sending a full tablet report...
I20260929 01:54:35.719048 24310 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "c70db43b7ed24252857d7e50932909bd" instance_seqno: 1790646874645094) as {username='kudu', principal='kudu/127.0.71.67@KRBTEST.COM'} at 127.0.71.67:51085; Asking this server to re-register.
I20260929 01:54:35.720276 24268 heartbeater.cc:461] Registering TS with master...
I20260929 01:54:35.722311 24310 ts_manager.cc:194] Registered new tserver with Master: c70db43b7ed24252857d7e50932909bd (127.0.71.67:38217)
W20260929 01:54:35.854691 24368 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:54:35.855065 24368 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:54:35.855228 24368 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:54:35.863965 24368 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:54:35.864243 24368 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:54:35.875159 24368 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:33083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=44615
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:54:35.876984 24368 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:54:35.878301 24368 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:54:35.888211 24375 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:35.888254 24374 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:35.889310 24377 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:54:35.889671 24368 server_base.cc:1061] running on GCE node
I20260929 01:54:35.890254 24368 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:54:35.890445 24368 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:54:35.890534 24368 hybrid_clock.cc:648] HybridClock initialized: now 1790646875890499 us; error 0 us; skew 500 ppm
I20260929 01:54:35.890816 24368 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:54:35 dist-test-slave-vz23 krb5kdc[23383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:54:35.894459 24368 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:54:35.896147 24368 webserver.cc:533] Webserver started at http://127.0.71.65:44615/ using document root <none> and password file <none>
I20260929 01:54:35.896921 24368 fs_manager.cc:362] Metadata directory not provided
I20260929 01:54:35.897059 24368 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:54:35.902558 24368 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:54:35.905318 24383 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:35.906610 24368 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:54:35.906788 24368 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "afc2290d563b4743b87fac3c7c266de6"
format_stamp: "Formatted at 2026-09-29 01:54:14 on dist-test-slave-vz23"
I20260929 01:54:35.907475 24368 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:54:35.933955 24368 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:54:35.937866 24368 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:54:35.938141 24368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:54:35.939098 24368 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:54:35.940755 24368 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:54:35.940881 24368 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:35.941005 24368 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:54:35.941102 24368 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:35.977295 24368 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:33083
I20260929 01:54:35.977360 24495 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:33083 every 8 connection(s)
I20260929 01:54:35.978869 24368 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:54:35.988270 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 24368
Sep 29 01:54:35 dist-test-slave-vz23 krb5kdc[23383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646861, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for hive/127.0.0.1@KRBTEST.COM
Sep 29 01:54:35 dist-test-slave-vz23 krb5kdc[23383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:54:35 dist-test-slave-vz23 krb5kdc[23383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:54:36.010641 24497 heartbeater.cc:344] Connected to a master server at 127.0.71.126:36771
I20260929 01:54:36.010779 24498 heartbeater.cc:344] Connected to a master server at 127.0.71.125:46761
I20260929 01:54:36.011154 24497 heartbeater.cc:461] Registering TS with master...
I20260929 01:54:36.011159 24498 heartbeater.cc:461] Registering TS with master...
I20260929 01:54:36.012506 24497 heartbeater.cc:507] Master 127.0.71.126:36771 requested a full tablet report, sending...
I20260929 01:54:36.012531 24498 heartbeater.cc:507] Master 127.0.71.125:46761 requested a full tablet report, sending...
I20260929 01:54:36.014607 24310 ts_manager.cc:194] Registered new tserver with Master: afc2290d563b4743b87fac3c7c266de6 (127.0.71.65:33083)
Sep 29 01:54:36 dist-test-slave-vz23 krb5kdc[23383](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646875, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:54:36.014781 23692 ts_manager.cc:194] Re-registered known tserver with Master: afc2290d563b4743b87fac3c7c266de6 (127.0.71.65:33083)
I20260929 01:54:36.022931 24496 heartbeater.cc:344] Connected to a master server at 127.0.71.124:45059
I20260929 01:54:36.023119 24496 heartbeater.cc:461] Registering TS with master...
I20260929 01:54:36.023603 24496 heartbeater.cc:507] Master 127.0.71.124:45059 requested a full tablet report, sending...
I20260929 01:54:36.025147 23782 ts_manager.cc:194] Re-registered known tserver with Master: afc2290d563b4743b87fac3c7c266de6 (127.0.71.65:33083)
I20260929 01:54:36.046768 24506 raft_consensus.cc:493] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7f6befd994534bbebdb63da555aabb73)
I20260929 01:54:36.047055 24506 raft_consensus.cc:515] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } }
I20260929 01:54:36.049551 24506 leader_election.cc:290] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f6befd994534bbebdb63da555aabb73 (127.0.71.126:36771), 84706cb3b1a64ebc9710b0c330b6ea73 (127.0.71.124:45059)
I20260929 01:54:36.062909 24320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ff24fc03c4e74eae9c1472eaa7875378" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7f6befd994534bbebdb63da555aabb73" is_pre_election: true
I20260929 01:54:36.063565 24320 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ff24fc03c4e74eae9c1472eaa7875378 in term 2.
I20260929 01:54:36.063596 23792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ff24fc03c4e74eae9c1472eaa7875378" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" is_pre_election: true
I20260929 01:54:36.064262 23680 leader_election.cc:304] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7f6befd994534bbebdb63da555aabb73, ff24fc03c4e74eae9c1472eaa7875378; no voters:
I20260929 01:54:36.064725 24506 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 01:54:36.064891 24506 raft_consensus.cc:493] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [term 2 FOLLOWER]: Starting leader election (detected failure of leader 7f6befd994534bbebdb63da555aabb73)
I20260929 01:54:36.065011 24506 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:54:36.068053 24506 raft_consensus.cc:515] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } }
I20260929 01:54:36.068897 24506 leader_election.cc:290] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [CANDIDATE]: Term 3 election: Requested vote from peers 7f6befd994534bbebdb63da555aabb73 (127.0.71.126:36771), 84706cb3b1a64ebc9710b0c330b6ea73 (127.0.71.124:45059)
I20260929 01:54:36.069707 24320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ff24fc03c4e74eae9c1472eaa7875378" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7f6befd994534bbebdb63da555aabb73"
I20260929 01:54:36.069754 23792 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ff24fc03c4e74eae9c1472eaa7875378" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "84706cb3b1a64ebc9710b0c330b6ea73"
I20260929 01:54:36.069923 24320 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:54:36.075680 24320 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ff24fc03c4e74eae9c1472eaa7875378 in term 3.
I20260929 01:54:36.076331 23680 leader_election.cc:304] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7f6befd994534bbebdb63da555aabb73, ff24fc03c4e74eae9c1472eaa7875378; no voters: 84706cb3b1a64ebc9710b0c330b6ea73
I20260929 01:54:36.076759 24506 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 01:54:36.077346 24506 raft_consensus.cc:697] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [term 3 LEADER]: Becoming Leader. State: Replica: ff24fc03c4e74eae9c1472eaa7875378, State: Running, Role: LEADER
I20260929 01:54:36.077898 24506 consensus_queue.cc:237] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } }
I20260929 01:54:36.080713 24509 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff24fc03c4e74eae9c1472eaa7875378. Latest consensus state: current_term: 3 leader_uuid: "ff24fc03c4e74eae9c1472eaa7875378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:36.081023 24509 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [sys.catalog]: This master's current role is: LEADER
I20260929 01:54:36.081780 24511 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:54:36.084434 24511 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:54:36.085565 24511 catalog_manager.cc:1308] Loaded cluster ID: 4b631c8567cf41bab15ecc52fd208f7f
I20260929 01:54:36.085714 24511 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:54:36.087414 24511 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:54:36.088529 24511 catalog_manager.cc:6230] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378: Loaded TSK: 0
I20260929 01:54:36.089730 24511 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:54:36.109104 24127 heartbeater.cc:344] Connected to a master server at 127.0.71.126:36771
I20260929 01:54:36.109360 24127 heartbeater.cc:499] Master 127.0.71.126:36771 was elected leader, sending a full tablet report...
I20260929 01:54:36.110334 24310 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "f95e349e0c094c5ba984109acf0ff50b" instance_seqno: 1790646874042245) as {username='kudu', principal='kudu/127.0.71.66@KRBTEST.COM'} at 127.0.71.66:38009; Asking this server to re-register.
I20260929 01:54:36.110984 24127 heartbeater.cc:461] Registering TS with master...
I20260929 01:54:36.112139 24310 ts_manager.cc:194] Registered new tserver with Master: f95e349e0c094c5ba984109acf0ff50b (127.0.71.66:35595)
I20260929 01:54:36.136638 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 24368
W20260929 01:54:36.137879 24516 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:54:36.134404 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:55764 (local address 127.0.71.65:33083)
0929 01:54:36.134992 (+ 588us) server_negotiation.cc:207] Beginning negotiation
0929 01:54:36.134999 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0929 01:54:36.135031 (+ 32us) server_negotiation.cc:408] Connection header received
0929 01:54:36.135129 (+ 98us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:54:36.135136 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:54:36.135340 (+ 204us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:54:36.135523 (+ 183us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:54:36.136428 (+ 905us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55764: BlockingRecv error: recv got EOF from 127.0.0.1:55764 (error 108)
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":295,"threads_started":1}
W20260929 01:54:36.139183 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 24368: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:33083: server requires authentication, but client does not have Kerberos credentials available
W20260929 01:54:36.139263 24516 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:54:36.137554 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:55766 (local address 127.0.71.65:33083)
0929 01:54:36.138213 (+ 659us) server_negotiation.cc:207] Beginning negotiation
0929 01:54:36.138221 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0929 01:54:36.138242 (+ 21us) server_negotiation.cc:408] Connection header received
0929 01:54:36.138283 (+ 41us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:54:36.138287 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:54:36.138335 (+ 48us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:54:36.138445 (+ 110us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:54:36.139090 (+ 645us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55766: BlockingRecv error: recv got EOF from 127.0.0.1:55766 (error 108)
Metrics: {"server-negotiator.queue_time_us":556,"thread_start_us":377,"threads_started":1}
I20260929 01:54:36.139355 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 24368
I20260929 01:54:36.156208 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:33083
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--webserver_port=44615
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:54:36.318450 24519 raft_consensus.cc:493] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader 7f6befd994534bbebdb63da555aabb73)
I20260929 01:54:36.318715 24519 raft_consensus.cc:515] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } }
I20260929 01:54:36.320710 24519 leader_election.cc:290] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers 7f6befd994534bbebdb63da555aabb73 (127.0.71.126:36771), ff24fc03c4e74eae9c1472eaa7875378 (127.0.71.125:46761)
I20260929 01:54:36.321617 23703 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "ff24fc03c4e74eae9c1472eaa7875378" is_pre_election: true
I20260929 01:54:36.327234 24320 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "7f6befd994534bbebdb63da555aabb73" is_pre_election: true
I20260929 01:54:36.327591 24320 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 84706cb3b1a64ebc9710b0c330b6ea73 in current term 3: Already voted for candidate ff24fc03c4e74eae9c1472eaa7875378 in this term.
I20260929 01:54:36.328555 23769 leader_election.cc:304] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 84706cb3b1a64ebc9710b0c330b6ea73; no voters: 7f6befd994534bbebdb63da555aabb73, ff24fc03c4e74eae9c1472eaa7875378
I20260929 01:54:36.328982 24519 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:54:36.331106 24519 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 01:54:36.501327 23792 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [term 3 FOLLOWER]: Refusing update from remote peer ff24fc03c4e74eae9c1472eaa7875378: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 01:54:36.502147 24509 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:54:36.504014 24519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff24fc03c4e74eae9c1472eaa7875378. Latest consensus state: current_term: 3 leader_uuid: "ff24fc03c4e74eae9c1472eaa7875378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:36.504276 24519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:54:36.506099 24506 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "ff24fc03c4e74eae9c1472eaa7875378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:36.506353 24506 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [sys.catalog]: This master's current role is: LEADER
I20260929 01:54:36.507261 24519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "ff24fc03c4e74eae9c1472eaa7875378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:36.507462 24519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 84706cb3b1a64ebc9710b0c330b6ea73 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:54:36.507493 24320 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [term 3 FOLLOWER]: Refusing update from remote peer ff24fc03c4e74eae9c1472eaa7875378: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 01:54:36.508329 24509 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260929 01:54:36.508598 24518 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:54:36.508951 24518 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:54:36.509114 24518 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:54:36.509238 24518 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260929 01:54:36.512920 24531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "ff24fc03c4e74eae9c1472eaa7875378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:36.513276 24531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:54:36.513810 24528 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ff24fc03c4e74eae9c1472eaa7875378. Latest consensus state: current_term: 3 leader_uuid: "ff24fc03c4e74eae9c1472eaa7875378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:36.514019 24528 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7f6befd994534bbebdb63da555aabb73 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:54:36.518910 24518 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:54:36.519181 24518 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:54:36.530905 24518 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:33083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=44615
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:54:36.532622 24518 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:54:36.533955 24518 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:54:36.544024 24534 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:36.544090 24533 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:36.544849 24536 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:54:36.545660 24518 server_base.cc:1061] running on GCE node
I20260929 01:54:36.546307 24518 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:54:36.546437 24518 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:54:36.546515 24518 hybrid_clock.cc:648] HybridClock initialized: now 1790646876546483 us; error 0 us; skew 500 ppm
I20260929 01:54:36.546746 24518 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:54:36 dist-test-slave-vz23 krb5kdc[23383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646876, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:54:36.550727 24518 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:54:36.552508 24518 webserver.cc:533] Webserver started at http://127.0.71.65:44615/ using document root <none> and password file <none>
I20260929 01:54:36.553328 24518 fs_manager.cc:362] Metadata directory not provided
I20260929 01:54:36.553442 24518 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:54:36.559470 24518 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 01:54:36.562505 24542 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:36.564071 24518 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:54:36.564271 24518 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "afc2290d563b4743b87fac3c7c266de6"
format_stamp: "Formatted at 2026-09-29 01:54:14 on dist-test-slave-vz23"
I20260929 01:54:36.565054 24518 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:54:36.593874 24518 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:54:36.613617 24518 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:54:36.613945 24518 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:54:36.614980 24518 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:54:36.616653 24518 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:54:36.616762 24518 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:36.616915 24518 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:54:36.617007 24518 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:54:36.617389 24518 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:54:36.617477 24518 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:54:36.617563 24518 fault_injection.cc:59] injecting fault for kudu (pid 24518): FLAGS_fault_before_start (process will exit)
I20260929 01:54:36.641746 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:33083
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--webserver_port=44615
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:54:36.978921 24527 sys_catalog.cc:455] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "ff24fc03c4e74eae9c1472eaa7875378" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7f6befd994534bbebdb63da555aabb73" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36771 } } peers { permanent_uuid: "ff24fc03c4e74eae9c1472eaa7875378" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46761 } } peers { permanent_uuid: "84706cb3b1a64ebc9710b0c330b6ea73" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45059 } } }
I20260929 01:54:36.979192 24527 sys_catalog.cc:458] T 00000000000000000000000000000000 P ff24fc03c4e74eae9c1472eaa7875378 [sys.catalog]: This master's current role is: LEADER
W20260929 01:54:36.991768 24550 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:54:36.992177 24550 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:54:36.992331 24550 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:54:37.001133 24550 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:54:37.001386 24550 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:54:37.016686 24550 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:33083
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=44615
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:36771,127.0.71.125:46761,127.0.71.124:45059
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:54:37.018390 24550 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:54:37.019981 24550 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:54:37.030326 24556 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:37.030305 24558 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:37.030436 24555 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:54:37.031498 24550 server_base.cc:1061] running on GCE node
I20260929 01:54:37.032131 24550 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:54:37.032334 24550 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:54:37.032442 24550 hybrid_clock.cc:648] HybridClock initialized: now 1790646877032416 us; error 0 us; skew 500 ppm
I20260929 01:54:37.032799 24550 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:54:37 dist-test-slave-vz23 krb5kdc[23383](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646877, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:54:37.036755 24550 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:54:37.038523 24550 webserver.cc:533] Webserver started at http://127.0.71.65:44615/ using document root <none> and password file <none>
I20260929 01:54:37.039258 24550 fs_manager.cc:362] Metadata directory not provided
I20260929 01:54:37.039366 24550 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:54:37.045239 24550 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:54:37.048372 24564 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:37.050058 24550 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:54:37.050227 24550 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "afc2290d563b4743b87fac3c7c266de6"
format_stamp: "Formatted at 2026-09-29 01:54:14 on dist-test-slave-vz23"
I20260929 01:54:37.051097 24550 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:54:37.071830 24550 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:54:37.076267 24550 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:54:37.076602 24550 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:54:37.077639 24550 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:54:37.079289 24550 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:54:37.079377 24550 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:37.079540 24550 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:54:37.079636 24550 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:37.094470 24128 heartbeater.cc:499] Master 127.0.71.125:46761 was elected leader, sending a full tablet report...
I20260929 01:54:37.124760 24550 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:33083
I20260929 01:54:37.124979 24676 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:33083 every 8 connection(s)
I20260929 01:54:37.126374 24550 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:54:37.129642 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 24550
I20260929 01:54:37.129977 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 24550
W20260929 01:54:37.147140 24686 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:54:37.141444 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:55768 (local address 127.0.71.65:33083)
0929 01:54:37.142177 (+ 733us) server_negotiation.cc:207] Beginning negotiation
0929 01:54:37.142186 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0929 01:54:37.142229 (+ 43us) server_negotiation.cc:408] Connection header received
0929 01:54:37.142433 (+ 204us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:54:37.142460 (+ 27us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:54:37.142806 (+ 346us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:54:37.143080 (+ 274us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:54:37.145424 (+ 2344us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55768: BlockingRecv error: recv got EOF from 127.0.0.1:55768 (error 108)
Metrics: {"server-negotiator.queue_time_us":588,"thread_start_us":419,"threads_started":1}
W20260929 01:54:37.152689 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 24550: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:33083: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:54:37.152902 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 24550
I20260929 01:54:37.169952 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 23996
W20260929 01:54:37.170639 24688 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:54:37.142811 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:49799 (local address 127.0.71.124:45059)
0929 01:54:37.158469 (+ 15658us) server_negotiation.cc:207] Beginning negotiation
0929 01:54:37.158478 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0929 01:54:37.169690 (+ 11212us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:49799: BlockingRecv error: recv got EOF from 127.0.71.65:49799 (error 108)
Metrics: {"server-negotiator.queue_time_us":15483,"thread_start_us":15315,"threads_started":1}
W20260929 01:54:37.170732 24684 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:54:37.132057 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:52747 (local address 127.0.71.125:46761)
0929 01:54:37.139964 (+ 7907us) server_negotiation.cc:207] Beginning negotiation
0929 01:54:37.139970 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0929 01:54:37.140250 (+ 280us) server_negotiation.cc:408] Connection header received
0929 01:54:37.148909 (+ 8659us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:54:37.148920 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:54:37.149041 (+ 121us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:54:37.149165 (+ 124us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:54:37.150388 (+ 1223us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:54:37.152238 (+ 1850us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:54:37.169859 (+ 17621us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:52747: BlockingRecv error: recv got EOF from 127.0.71.65:52747 (error 108)
Metrics: {"server-negotiator.queue_time_us":7766,"thread_start_us":8925,"threads_started":1}
W20260929 01:54:37.170732 24681 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:54:37.130349 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:53741 (local address 127.0.71.126:36771)
0929 01:54:37.136873 (+ 6524us) server_negotiation.cc:207] Beginning negotiation
0929 01:54:37.136878 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0929 01:54:37.140976 (+ 4098us) server_negotiation.cc:408] Connection header received
0929 01:54:37.142658 (+ 1682us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:54:37.142669 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:54:37.142787 (+ 118us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:54:37.142939 (+ 152us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:54:37.146280 (+ 3341us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:54:37.148202 (+ 1922us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:54:37.150841 (+ 2639us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:54:37.151518 (+ 677us) server_negotiation.cc:658] Negotiated auth-only TLSv1.3 with cipher TLS_AES_128_GCM_SHA256 TLSv1.3 Kx=any Au=any Enc=AESGCM(128) Mac=AEAD
0929 01:54:37.169893 (+ 18375us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:53741: BlockingRecv error: recv got EOF from 127.0.71.65:53741 (error 108)
Metrics: {"server-negotiator.queue_time_us":6399,"thread_start_us":6286,"threads_started":1}
W20260929 01:54:37.173418 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 23996: Not authorized: Client connection negotiation failed: client connection to 127.0.71.66:35595: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:54:37.173595 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 23996
I20260929 01:54:37.189769 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 24140
W20260929 01:54:37.192602 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 24140: Not authorized: Client connection negotiation failed: client connection to 127.0.71.67:38217: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:54:37.192797 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 24140
I20260929 01:54:37.209088 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 24278
W20260929 01:54:37.211342 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 24278: Not authorized: Client connection negotiation failed: client connection to 127.0.71.126:36771: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:54:37.211488 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 24278
W20260929 01:54:37.211510 24681 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:54:37.210094 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:54870 (local address 127.0.71.126:36771)
0929 01:54:37.210269 (+ 175us) server_negotiation.cc:207] Beginning negotiation
0929 01:54:37.210273 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0929 01:54:37.210290 (+ 17us) server_negotiation.cc:408] Connection header received
0929 01:54:37.210415 (+ 125us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:54:37.210419 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:54:37.210484 (+ 65us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:54:37.210570 (+ 86us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:54:37.211273 (+ 703us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54870: BlockingRecv error: recv got EOF from 127.0.0.1:54870 (error 108)
Metrics: {"server-negotiator.queue_time_us":60}
I20260929 01:54:37.231773 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 23662
I20260929 01:54:37.233088 23723 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:54:37.347030 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 23662
I20260929 01:54:37.369251 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 23751
I20260929 01:54:37.370560 23812 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:54:37.484385 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 23751
Shutting down hive metastore.
[ OK ] ExternalMiniClusterTest.TestBasicOperation/9 (44304 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/10
Loading random data
Initializing database '/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:54:37 dist-test-slave-vz23 krb5kdc[24700](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:54:37 dist-test-slave-vz23 krb5kdc[24700](info): set up 2 sockets
Sep 29 01:54:37 dist-test-slave-vz23 krb5kdc[24700](info): commencing operation
krb5kdc: starting...
W20260929 01:54:39.965370 285 mini_kdc.cc:121] Time spent starting KDC: real 2.058s user 0.001s sys 0.024s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 01:54:40 dist-test-slave-vz23 krb5kdc[24700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:54:40Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:54:40Z Disabled control of system clock
WARNING: no policy specified for hive/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "hive/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/hive_127.0.0.1.keytab.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/hive_127.0.0.1.keytab.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5.conf
2026-09-29 01:54:48: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
Sep 29 01:54:50 dist-test-slave-vz23 krb5kdc[24700](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790646890, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 01:54:56 dist-test-slave-vz23 krb5kdc[24700](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790646896, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 01:54:56 dist-test-slave-vz23 krb5kdc[24700](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790646896, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:54:58.932260 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:41391
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:35193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:54:59.294025 24996 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:54:59.294381 24996 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:54:59.294579 24996 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:54:59.303504 24996 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:54:59.303627 24996 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:54:59.303736 24996 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:54:59.303833 24996 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:54:59.315407 24996 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:41391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:54:59.317188 24996 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:54:59.318559 24996 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:54:59.327883 25002 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:59.327935 25001 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:59.327977 25004 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:54:59.328900 24996 server_base.cc:1061] running on GCE node
I20260929 01:54:59.329876 24996 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:54:59.331655 24996 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:54:59.332916 24996 hybrid_clock.cc:648] HybridClock initialized: now 1790646899332870 us; error 54 us; skew 500 ppm
I20260929 01:54:59.333302 24996 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:54:59 dist-test-slave-vz23 krb5kdc[24700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:54:59.337136 24996 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:54:59.339036 24996 webserver.cc:533] Webserver started at http://127.0.71.126:42299/ using document root <none> and password file <none>
I20260929 01:54:59.339813 24996 fs_manager.cc:362] Metadata directory not provided
I20260929 01:54:59.339957 24996 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:54:59.340317 24996 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:54:59.342983 24996 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "d6cf88e894de42e49d440d3b67403d37"
format_stamp: "Formatted at 2026-09-29 01:54:59 on dist-test-slave-vz23"
I20260929 01:54:59.343843 24996 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "d6cf88e894de42e49d440d3b67403d37"
format_stamp: "Formatted at 2026-09-29 01:54:59 on dist-test-slave-vz23"
I20260929 01:54:59.349766 24996 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.001s
I20260929 01:54:59.353825 25011 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:59.355371 24996 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 01:54:59.355571 24996 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "d6cf88e894de42e49d440d3b67403d37"
format_stamp: "Formatted at 2026-09-29 01:54:59 on dist-test-slave-vz23"
I20260929 01:54:59.355789 24996 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:54:59.381330 24996 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:54:59.398761 24996 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:54:59.399216 24996 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:54:59.420290 24996 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:41391
I20260929 01:54:59.420321 25062 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:41391 every 8 connection(s)
I20260929 01:54:59.421917 24996 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:54:59.426409 25063 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:54:59.427908 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 24996
I20260929 01:54:59.428134 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:54:59.432981 25063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:54:59 dist-test-slave-vz23 krb5kdc[24700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.125@KRBTEST.COM" created.
I20260929 01:54:59.460147 25063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:54:59.461687 25013 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.125:35913: connect: Connection refused (error 111)
W20260929 01:54:59.463397 25063 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:35913: Network error: Client connection negotiation failed: client connection to 127.0.71.125:35913: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.125@KRBTEST.COM" created.
I20260929 01:54:59.512832 25063 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } attempt: 1
W20260929 01:54:59.515306 25063 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:35913: Network error: Client connection negotiation failed: client connection to 127.0.71.125:35913: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:54:59.536602 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:35913
--webserver_interface=127.0.71.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:35193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:54:59.583630 25063 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } attempt: 2
W20260929 01:54:59.585853 25063 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:35913: Network error: Client connection negotiation failed: client connection to 127.0.71.125:35913: connect: Connection refused (error 111)
I20260929 01:54:59.677227 25063 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } attempt: 3
W20260929 01:54:59.679418 25063 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:35913: Network error: Client connection negotiation failed: client connection to 127.0.71.125:35913: connect: Connection refused (error 111)
I20260929 01:54:59.822737 25063 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } attempt: 4
W20260929 01:54:59.824981 25063 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:35913: Network error: Client connection negotiation failed: client connection to 127.0.71.125:35913: connect: Connection refused (error 111)
W20260929 01:54:59.877102 25072 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:54:59.877465 25072 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:54:59.877553 25072 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:54:59.886193 25072 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:54:59.886317 25072 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:54:59.886384 25072 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:54:59.886432 25072 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:54:59.897701 25072 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:35913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:54:59.899343 25072 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:54:59.900709 25072 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:54:59.910377 25078 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:59.910451 25080 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:54:59.910422 25077 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:54:59.910660 25072 server_base.cc:1061] running on GCE node
I20260929 01:54:59.911996 25072 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:54:59.913695 25072 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:54:59.914949 25072 hybrid_clock.cc:648] HybridClock initialized: now 1790646899914907 us; error 44 us; skew 500 ppm
I20260929 01:54:59.915290 25072 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:54:59 dist-test-slave-vz23 krb5kdc[24700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:54:59.918720 25072 init.cc:377] Logged in from keytab as kudu/127.0.71.125@KRBTEST.COM (short username kudu)
I20260929 01:54:59.920603 25072 webserver.cc:533] Webserver started at http://127.0.71.125:41801/ using document root <none> and password file <none>
I20260929 01:54:59.921357 25072 fs_manager.cc:362] Metadata directory not provided
I20260929 01:54:59.921458 25072 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:54:59.921784 25072 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:54:59.924714 25072 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/data/instance:
uuid: "38f49999d352471d8b1571385b4363c5"
format_stamp: "Formatted at 2026-09-29 01:54:59 on dist-test-slave-vz23"
I20260929 01:54:59.925647 25072 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/wal/instance:
uuid: "38f49999d352471d8b1571385b4363c5"
format_stamp: "Formatted at 2026-09-29 01:54:59 on dist-test-slave-vz23"
I20260929 01:54:59.932349 25072 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 01:54:59.936017 25087 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:54:59.937623 25072 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:54:59.937811 25072 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "38f49999d352471d8b1571385b4363c5"
format_stamp: "Formatted at 2026-09-29 01:54:59 on dist-test-slave-vz23"
I20260929 01:54:59.938000 25072 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:54:59.953418 25072 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:54:59.957705 25072 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:54:59.958042 25072 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:54:59.980208 25072 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:35913
I20260929 01:54:59.980211 25138 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:35913 every 8 connection(s)
I20260929 01:54:59.981824 25072 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:54:59.986488 25139 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:54:59.991197 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 25072
I20260929 01:54:59.991479 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:54:59.992898 25139 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:55:00 dist-test-slave-vz23 krb5kdc[24700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.124@KRBTEST.COM" created.
I20260929 01:55:00.017967 25139 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:55:00 dist-test-slave-vz23 krb5kdc[24700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:55:00.036764 25139 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:55:00.038311 25088 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.124:32895: connect: Connection refused (error 111)
W20260929 01:55:00.040262 25139 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:32895: Network error: Client connection negotiation failed: client connection to 127.0.71.124:32895: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.124@KRBTEST.COM" created.
I20260929 01:55:00.089682 25139 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } attempt: 1
W20260929 01:55:00.091924 25139 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:32895: Network error: Client connection negotiation failed: client connection to 127.0.71.124:32895: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:55:00.104815 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:32895
--webserver_interface=127.0.71.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:35193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:55:00.124294 25063 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } attempt: 5
Sep 29 01:55:00 dist-test-slave-vz23 krb5kdc[24700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:55:00.139456 25063 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:55:00.141731 25063 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:32895: Network error: Client connection negotiation failed: client connection to 127.0.71.124:32895: connect: Connection refused (error 111)
I20260929 01:55:00.160207 25139 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } attempt: 2
W20260929 01:55:00.162504 25139 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:32895: Network error: Client connection negotiation failed: client connection to 127.0.71.124:32895: connect: Connection refused (error 111)
I20260929 01:55:00.193012 25063 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } attempt: 1
W20260929 01:55:00.195276 25063 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:32895: Network error: Client connection negotiation failed: client connection to 127.0.71.124:32895: connect: Connection refused (error 111)
I20260929 01:55:00.253798 25139 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } attempt: 3
W20260929 01:55:00.256374 25139 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:32895: Network error: Client connection negotiation failed: client connection to 127.0.71.124:32895: connect: Connection refused (error 111)
I20260929 01:55:00.263602 25063 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } attempt: 2
W20260929 01:55:00.266039 25063 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:32895: Network error: Client connection negotiation failed: client connection to 127.0.71.124:32895: connect: Connection refused (error 111)
I20260929 01:55:00.372316 25063 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } attempt: 3
W20260929 01:55:00.374655 25063 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:32895: Network error: Client connection negotiation failed: client connection to 127.0.71.124:32895: connect: Connection refused (error 111)
I20260929 01:55:00.399718 25139 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } attempt: 4
W20260929 01:55:00.402109 25139 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:32895: Network error: Client connection negotiation failed: client connection to 127.0.71.124:32895: connect: Connection refused (error 111)
W20260929 01:55:00.454914 25149 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:55:00.455258 25149 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:55:00.455343 25149 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:55:00.464082 25149 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:55:00.464207 25149 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:55:00.464272 25149 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:55:00.464318 25149 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:55:00.475622 25149 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:32895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:55:00.477313 25149 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:55:00.478621 25149 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:55:00.487701 25154 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:00.487964 25155 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:00.488190 25157 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:00.489125 25149 server_base.cc:1061] running on GCE node
I20260929 01:55:00.490027 25149 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:55:00.491632 25149 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:55:00.492906 25149 hybrid_clock.cc:648] HybridClock initialized: now 1790646900492835 us; error 83 us; skew 500 ppm
I20260929 01:55:00.493323 25149 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:55:00 dist-test-slave-vz23 krb5kdc[24700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:55:00.497047 25149 init.cc:377] Logged in from keytab as kudu/127.0.71.124@KRBTEST.COM (short username kudu)
I20260929 01:55:00.498827 25149 webserver.cc:533] Webserver started at http://127.0.71.124:32859/ using document root <none> and password file <none>
I20260929 01:55:00.499675 25149 fs_manager.cc:362] Metadata directory not provided
I20260929 01:55:00.499778 25149 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:55:00.500082 25149 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:55:00.502904 25149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/data/instance:
uuid: "978f92c285a44324bb56bc9bf17fbce0"
format_stamp: "Formatted at 2026-09-29 01:55:00 on dist-test-slave-vz23"
I20260929 01:55:00.503743 25149 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/wal/instance:
uuid: "978f92c285a44324bb56bc9bf17fbce0"
format_stamp: "Formatted at 2026-09-29 01:55:00 on dist-test-slave-vz23"
I20260929 01:55:00.510257 25149 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.008s sys 0.000s
I20260929 01:55:00.514034 25164 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:00.515558 25149 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:55:00.515727 25149 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "978f92c285a44324bb56bc9bf17fbce0"
format_stamp: "Formatted at 2026-09-29 01:55:00 on dist-test-slave-vz23"
I20260929 01:55:00.515916 25149 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:55:00.532277 25149 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:55:00.536638 25149 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:55:00.536962 25149 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:55:00.551937 25063 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } attempt: 4
W20260929 01:55:00.554399 25063 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:32895: Network error: Client connection negotiation failed: client connection to 127.0.71.124:32895: connect: Connection refused (error 111)
I20260929 01:55:00.560668 25149 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:32895
I20260929 01:55:00.560688 25215 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:32895 every 8 connection(s)
I20260929 01:55:00.562573 25149 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:55:00.567003 25216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:55:00.570726 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 25149
I20260929 01:55:00.570966 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:55:00.573544 25216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:55:00 dist-test-slave-vz23 krb5kdc[24700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:55:00 dist-test-slave-vz23 krb5kdc[24700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:55:00.597707 25216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:55:00 dist-test-slave-vz23 krb5kdc[24700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:55:00 dist-test-slave-vz23 krb5kdc[24700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:55:00.613242 25216 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:55:00 dist-test-slave-vz23 krb5kdc[24700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:55:00 dist-test-slave-vz23 krb5kdc[24700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646900, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:55:00.632854 25216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0: Bootstrap starting.
I20260929 01:55:00.636329 25216 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0: Neither blocks nor log segments found. Creating new log.
I20260929 01:55:00.637827 25216 log.cc:826] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0: Log is configured to *not* fsync() on all Append() calls
I20260929 01:55:00.640372 25216 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0: No bootstrap required, opened a new log
I20260929 01:55:00.646268 25216 raft_consensus.cc:359] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } }
I20260929 01:55:00.646585 25216 raft_consensus.cc:385] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:55:00.646739 25216 raft_consensus.cc:740] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 978f92c285a44324bb56bc9bf17fbce0, State: Initialized, Role: FOLLOWER
I20260929 01:55:00.647456 25216 consensus_queue.cc:260] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } }
I20260929 01:55:00.648427 25226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:00.648960 25226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:55:00.649971 25216 sys_catalog.cc:565] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:55:00.661597 25237 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:55:00.661746 25237 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:55:00.658548 25149 master_runner.cc:187] Error getting master registration for 127.0.71.126:41391: OK, instance_id { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" instance_seqno: 1790646899414572 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:55:00.663391 25149 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:55:00.701406 25139 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } attempt: 5
Sep 29 01:55:00 dist-test-slave-vz23 krb5kdc[24700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:55:00.721731 25139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5: Bootstrap starting.
I20260929 01:55:00.726137 25139 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5: Neither blocks nor log segments found. Creating new log.
I20260929 01:55:00.727680 25139 log.cc:826] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5: Log is configured to *not* fsync() on all Append() calls
I20260929 01:55:00.730975 25139 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5: No bootstrap required, opened a new log
I20260929 01:55:00.738895 25139 raft_consensus.cc:359] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } }
I20260929 01:55:00.739280 25139 raft_consensus.cc:385] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:55:00.739410 25139 raft_consensus.cc:740] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38f49999d352471d8b1571385b4363c5, State: Initialized, Role: FOLLOWER
I20260929 01:55:00.740182 25139 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } }
I20260929 01:55:00.741250 25240 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:00.741525 25240 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:55:00.742700 25139 sys_catalog.cc:565] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:55:00.755259 25251 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:55:00.755515 25251 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:55:00.750341 25072 master_runner.cc:187] Error getting master registration for 127.0.71.126:41391: OK, instance_id { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" instance_seqno: 1790646899414572 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:55:00.757704 25072 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:55:00.789219 25170 tablet.cc:2433] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:55:00.831674 25063 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } attempt: 5
Sep 29 01:55:00 dist-test-slave-vz23 krb5kdc[24700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646899, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:55:00.857286 25063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: Bootstrap starting.
I20260929 01:55:00.862157 25063 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: Neither blocks nor log segments found. Creating new log.
I20260929 01:55:00.863780 25063 log.cc:826] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: Log is configured to *not* fsync() on all Append() calls
I20260929 01:55:00.867420 25063 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: No bootstrap required, opened a new log
I20260929 01:55:00.876344 25063 raft_consensus.cc:359] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } }
I20260929 01:55:00.876942 25063 raft_consensus.cc:385] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:55:00.877173 25063 raft_consensus.cc:740] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6cf88e894de42e49d440d3b67403d37, State: Initialized, Role: FOLLOWER
I20260929 01:55:00.878094 25063 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } }
I20260929 01:55:00.880916 25254 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:00.881264 25254 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:55:00.882489 25063 sys_catalog.cc:565] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:55:00.900117 24996 master_runner.cc:264] No leader master found from master d6cf88e894de42e49d440d3b67403d37
I20260929 01:55:00.900303 24996 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:55:00.902616 25017 tablet.cc:2433] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:55:00.904114 25265 catalog_manager.cc:1607] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:55:00.904299 25265 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 01:55:00.961016 25093 tablet.cc:2433] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:55:01.102092 25226 raft_consensus.cc:493] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:55:01.102445 25226 raft_consensus.cc:515] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } }
I20260929 01:55:01.104681 25226 leader_election.cc:290] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6cf88e894de42e49d440d3b67403d37 (127.0.71.126:41391), 38f49999d352471d8b1571385b4363c5 (127.0.71.125:35913)
I20260929 01:55:01.106592 25038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "978f92c285a44324bb56bc9bf17fbce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6cf88e894de42e49d440d3b67403d37" is_pre_election: true
I20260929 01:55:01.107215 25038 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 978f92c285a44324bb56bc9bf17fbce0 in term 0.
I20260929 01:55:01.108222 25240 raft_consensus.cc:493] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:55:01.109148 25167 leader_election.cc:304] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 978f92c285a44324bb56bc9bf17fbce0, d6cf88e894de42e49d440d3b67403d37; no voters:
I20260929 01:55:01.109848 25240 raft_consensus.cc:515] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } }
I20260929 01:55:01.111845 25240 leader_election.cc:290] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d6cf88e894de42e49d440d3b67403d37 (127.0.71.126:41391), 978f92c285a44324bb56bc9bf17fbce0 (127.0.71.124:32895)
I20260929 01:55:01.113639 25114 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "978f92c285a44324bb56bc9bf17fbce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38f49999d352471d8b1571385b4363c5" is_pre_election: true
I20260929 01:55:01.114054 25114 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 978f92c285a44324bb56bc9bf17fbce0 in term 0.
I20260929 01:55:01.115078 25191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38f49999d352471d8b1571385b4363c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "978f92c285a44324bb56bc9bf17fbce0" is_pre_election: true
I20260929 01:55:01.115515 25191 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38f49999d352471d8b1571385b4363c5 in term 0.
I20260929 01:55:01.116770 25038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38f49999d352471d8b1571385b4363c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6cf88e894de42e49d440d3b67403d37" is_pre_election: true
I20260929 01:55:01.117020 25038 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38f49999d352471d8b1571385b4363c5 in term 0.
I20260929 01:55:01.117821 25090 leader_election.cc:304] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38f49999d352471d8b1571385b4363c5, d6cf88e894de42e49d440d3b67403d37; no voters:
I20260929 01:55:01.119033 25240 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:55:01.119195 25240 raft_consensus.cc:493] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:55:01.119328 25240 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:55:01.125794 25226 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:55:01.126017 25226 raft_consensus.cc:493] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:55:01.126147 25226 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:55:01.132364 25240 raft_consensus.cc:515] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } }
I20260929 01:55:01.137524 25038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38f49999d352471d8b1571385b4363c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6cf88e894de42e49d440d3b67403d37"
I20260929 01:55:01.137807 25038 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:55:01.141970 25038 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38f49999d352471d8b1571385b4363c5 in term 1.
I20260929 01:55:01.142872 25090 leader_election.cc:304] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38f49999d352471d8b1571385b4363c5, d6cf88e894de42e49d440d3b67403d37; no voters:
I20260929 01:55:01.133756 25226 raft_consensus.cc:515] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } }
I20260929 01:55:01.147964 25038 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "978f92c285a44324bb56bc9bf17fbce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d6cf88e894de42e49d440d3b67403d37"
I20260929 01:55:01.148612 25038 raft_consensus.cc:2393] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 978f92c285a44324bb56bc9bf17fbce0 in current term 1: Already voted for candidate 38f49999d352471d8b1571385b4363c5 in this term.
I20260929 01:55:01.149796 25191 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38f49999d352471d8b1571385b4363c5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "978f92c285a44324bb56bc9bf17fbce0"
I20260929 01:55:01.150256 25191 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 38f49999d352471d8b1571385b4363c5 in current term 1: Already voted for candidate 978f92c285a44324bb56bc9bf17fbce0 in this term.
I20260929 01:55:01.152540 25240 leader_election.cc:290] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [CANDIDATE]: Term 1 election: Requested vote from peers d6cf88e894de42e49d440d3b67403d37 (127.0.71.126:41391), 978f92c285a44324bb56bc9bf17fbce0 (127.0.71.124:32895)
I20260929 01:55:01.154616 25114 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "978f92c285a44324bb56bc9bf17fbce0" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "38f49999d352471d8b1571385b4363c5"
I20260929 01:55:01.153595 25240 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:55:01.155737 25226 leader_election.cc:290] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [CANDIDATE]: Term 1 election: Requested vote from peers d6cf88e894de42e49d440d3b67403d37 (127.0.71.126:41391), 38f49999d352471d8b1571385b4363c5 (127.0.71.125:35913)
I20260929 01:55:01.156035 25240 raft_consensus.cc:697] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [term 1 LEADER]: Becoming Leader. State: Replica: 38f49999d352471d8b1571385b4363c5, State: Running, Role: LEADER
I20260929 01:55:01.157280 25240 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } }
I20260929 01:55:01.167706 25166 leader_election.cc:304] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 978f92c285a44324bb56bc9bf17fbce0; no voters: 38f49999d352471d8b1571385b4363c5, d6cf88e894de42e49d440d3b67403d37
I20260929 01:55:01.173066 25226 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 01:55:01.173923 25267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38f49999d352471d8b1571385b4363c5. Latest consensus state: current_term: 1 leader_uuid: "38f49999d352471d8b1571385b4363c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:01.174446 25267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [sys.catalog]: This master's current role is: LEADER
I20260929 01:55:01.181566 25269 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:55:01.184536 25269 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:55:01.194708 25191 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [term 1 FOLLOWER]: Refusing update from remote peer 38f49999d352471d8b1571385b4363c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:55:01.194706 25038 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [term 1 FOLLOWER]: Refusing update from remote peer 38f49999d352471d8b1571385b4363c5: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:55:01.195813 25267 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:55:01.196171 25240 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:55:01.225967 25254 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38f49999d352471d8b1571385b4363c5. Latest consensus state: current_term: 1 leader_uuid: "38f49999d352471d8b1571385b4363c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:01.226269 25254 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:55:01.226728 25270 mvcc.cc:204] Tried to move back new op lower bound from 7334489707279147008 to 7334489707157532672. Current Snapshot: MvccSnapshot[applied={T|T < 7334489707279147008}]
I20260929 01:55:01.229645 25254 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "38f49999d352471d8b1571385b4363c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:01.229876 25254 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:55:01.234553 25226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38f49999d352471d8b1571385b4363c5. Latest consensus state: current_term: 1 leader_uuid: "38f49999d352471d8b1571385b4363c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:01.234860 25226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:55:01.237466 25272 mvcc.cc:204] Tried to move back new op lower bound from 7334489707279147008 to 7334489707157532672. Current Snapshot: MvccSnapshot[applied={T|T < 7334489707279147008}]
I20260929 01:55:01.241828 25269 catalog_manager.cc:1396] Generated new cluster ID: 311e41733bad4ff691115e2dbb636f4d
I20260929 01:55:01.241957 25269 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:55:01.242537 25267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "38f49999d352471d8b1571385b4363c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:01.242810 25267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [sys.catalog]: This master's current role is: LEADER
I20260929 01:55:01.243289 25267 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "38f49999d352471d8b1571385b4363c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:01.243486 25267 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [sys.catalog]: This master's current role is: LEADER
I20260929 01:55:01.251390 25226 sys_catalog.cc:455] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "38f49999d352471d8b1571385b4363c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:01.251660 25226 sys_catalog.cc:458] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:55:01.274688 25269 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 01:55:01.276471 25269 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:55:01.296499 25269 catalog_manager.cc:6219] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5: Generated new TSK 0
I20260929 01:55:01.297648 25269 catalog_manager.cc:1563] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:55:01.436872 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:35193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:55:01.663739 25237 catalog_manager.cc:1308] Loaded cluster ID: 311e41733bad4ff691115e2dbb636f4d
I20260929 01:55:01.663890 25237 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0: loading cluster ID for follower catalog manager: success
I20260929 01:55:01.666707 25237 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0: acquiring CA information for follower catalog manager: success
I20260929 01:55:01.668707 25237 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 01:55:01.788915 25284 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:55:01.789240 25284 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:55:01.789408 25284 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:55:01.798403 25284 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:55:01.798668 25284 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:55:01.811244 25284 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:55:01.813055 25284 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:55:01.814404 25284 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:55:01.825241 25292 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:01.825273 25290 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:01.825626 25289 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:01.826195 25284 server_base.cc:1061] running on GCE node
I20260929 01:55:01.826906 25284 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:55:01.828056 25284 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:55:01.829312 25284 hybrid_clock.cc:648] HybridClock initialized: now 1790646901829214 us; error 101 us; skew 500 ppm
I20260929 01:55:01.829730 25284 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:55:01 dist-test-slave-vz23 krb5kdc[24700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:55:01.833289 25284 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:55:01.835305 25284 webserver.cc:533] Webserver started at http://127.0.71.65:44163/ using document root <none> and password file <none>
I20260929 01:55:01.836246 25284 fs_manager.cc:362] Metadata directory not provided
I20260929 01:55:01.836385 25284 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:55:01.836812 25284 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:55:01.839479 25284 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "bc92f4e26ca24156a850ac6f0c03209c"
format_stamp: "Formatted at 2026-09-29 01:55:01 on dist-test-slave-vz23"
I20260929 01:55:01.840301 25284 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "bc92f4e26ca24156a850ac6f0c03209c"
format_stamp: "Formatted at 2026-09-29 01:55:01 on dist-test-slave-vz23"
I20260929 01:55:01.846303 25284 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 01:55:01.850359 25300 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:01.852079 25284 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 01:55:01.852299 25284 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "bc92f4e26ca24156a850ac6f0c03209c"
format_stamp: "Formatted at 2026-09-29 01:55:01 on dist-test-slave-vz23"
I20260929 01:55:01.852581 25284 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:55:01.889981 25284 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:55:01.894598 25284 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:55:01.894929 25284 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:55:01.895953 25284 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:55:01.897667 25284 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:55:01.897751 25284 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:01.897903 25284 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:55:01.897966 25284 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:01.906083 25265 catalog_manager.cc:1308] Loaded cluster ID: 311e41733bad4ff691115e2dbb636f4d
I20260929 01:55:01.906178 25265 catalog_manager.cc:1601] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: loading cluster ID for follower catalog manager: success
I20260929 01:55:01.909896 25265 catalog_manager.cc:1623] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: acquiring CA information for follower catalog manager: success
I20260929 01:55:01.912573 25265 catalog_manager.cc:1651] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:55:01.936200 25284 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:38659
I20260929 01:55:01.936255 25412 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:38659 every 8 connection(s)
I20260929 01:55:01.937821 25284 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:55:01.945667 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 25284
I20260929 01:55:01.945945 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
Sep 29 01:55:01 dist-test-slave-vz23 krb5kdc[24700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:55:01 dist-test-slave-vz23 krb5kdc[24700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:55:01 dist-test-slave-vz23 krb5kdc[24700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646901, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.66@KRBTEST.COM" created.
I20260929 01:55:01.997032 25414 heartbeater.cc:344] Connected to a master server at 127.0.71.126:41391
I20260929 01:55:01.997144 25413 heartbeater.cc:344] Connected to a master server at 127.0.71.124:32895
I20260929 01:55:01.997500 25413 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:01.997496 25414 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:01.998575 25413 heartbeater.cc:507] Master 127.0.71.124:32895 requested a full tablet report, sending...
I20260929 01:55:01.998600 25414 heartbeater.cc:507] Master 127.0.71.126:41391 requested a full tablet report, sending...
I20260929 01:55:02.001087 25181 ts_manager.cc:194] Registered new tserver with Master: bc92f4e26ca24156a850ac6f0c03209c (127.0.71.65:38659)
I20260929 01:55:02.001123 25027 ts_manager.cc:194] Registered new tserver with Master: bc92f4e26ca24156a850ac6f0c03209c (127.0.71.65:38659)
I20260929 01:55:02.004626 25415 heartbeater.cc:344] Connected to a master server at 127.0.71.125:35913
I20260929 01:55:02.004777 25415 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:02.005312 25415 heartbeater.cc:507] Master 127.0.71.125:35913 requested a full tablet report, sending...
I20260929 01:55:02.007325 25104 ts_manager.cc:194] Registered new tserver with Master: bc92f4e26ca24156a850ac6f0c03209c (127.0.71.65:38659)
I20260929 01:55:02.009757 25104 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:55267
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:55:02.085810 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.66:0
--local_ip_for_outbound_sockets=127.0.71.66
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:35193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:55:02.439730 25427 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:55:02.440052 25427 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:55:02.440239 25427 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:55:02.449576 25427 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:55:02.449851 25427 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.66
I20260929 01:55:02.461350 25427 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.71.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:55:02.462987 25427 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:55:02.464533 25427 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:55:02.474663 25432 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:02.474611 25433 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:02.475440 25435 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:02.476068 25427 server_base.cc:1061] running on GCE node
I20260929 01:55:02.476780 25427 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:55:02.477880 25427 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:55:02.479103 25427 hybrid_clock.cc:648] HybridClock initialized: now 1790646902479042 us; error 56 us; skew 500 ppm
I20260929 01:55:02.479481 25427 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:55:02 dist-test-slave-vz23 krb5kdc[24700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:55:02.483579 25427 init.cc:377] Logged in from keytab as kudu/127.0.71.66@KRBTEST.COM (short username kudu)
I20260929 01:55:02.485519 25427 webserver.cc:533] Webserver started at http://127.0.71.66:38923/ using document root <none> and password file <none>
I20260929 01:55:02.486286 25427 fs_manager.cc:362] Metadata directory not provided
I20260929 01:55:02.486425 25427 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:55:02.486790 25427 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:55:02.489426 25427 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/data/instance:
uuid: "d36e7a7040c34330b7346caf16813ae2"
format_stamp: "Formatted at 2026-09-29 01:55:02 on dist-test-slave-vz23"
I20260929 01:55:02.490221 25427 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/wal/instance:
uuid: "d36e7a7040c34330b7346caf16813ae2"
format_stamp: "Formatted at 2026-09-29 01:55:02 on dist-test-slave-vz23"
I20260929 01:55:02.496065 25427 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.003s
I20260929 01:55:02.500072 25442 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:02.501657 25427 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.003s
I20260929 01:55:02.501865 25427 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/wal
uuid: "d36e7a7040c34330b7346caf16813ae2"
format_stamp: "Formatted at 2026-09-29 01:55:02 on dist-test-slave-vz23"
I20260929 01:55:02.502133 25427 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:55:02.515671 25427 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:55:02.519892 25427 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:55:02.520200 25427 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:55:02.521230 25427 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:55:02.522827 25427 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:55:02.522935 25427 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:02.523087 25427 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:55:02.523160 25427 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:02.560791 25427 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.66:45349
I20260929 01:55:02.560825 25554 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.66:45349 every 8 connection(s)
I20260929 01:55:02.562475 25427 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
I20260929 01:55:02.571357 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 25427
I20260929 01:55:02.571601 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/wal/instance
Sep 29 01:55:02 dist-test-slave-vz23 krb5kdc[24700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:55:02 dist-test-slave-vz23 krb5kdc[24700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:55:02 dist-test-slave-vz23 krb5kdc[24700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646902, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:55:02.609840 25556 heartbeater.cc:344] Connected to a master server at 127.0.71.126:41391
I20260929 01:55:02.610251 25556 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:02.611140 25557 heartbeater.cc:344] Connected to a master server at 127.0.71.125:35913
I20260929 01:55:02.611299 25557 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:02.611301 25556 heartbeater.cc:507] Master 127.0.71.126:41391 requested a full tablet report, sending...
I20260929 01:55:02.611891 25557 heartbeater.cc:507] Master 127.0.71.125:35913 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.71.67@KRBTEST.COM; defaulting to no policy
I20260929 01:55:02.613370 25555 heartbeater.cc:344] Connected to a master server at 127.0.71.124:32895
I20260929 01:55:02.613521 25555 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:02.613577 25027 ts_manager.cc:194] Registered new tserver with Master: d36e7a7040c34330b7346caf16813ae2 (127.0.71.66:45349)
I20260929 01:55:02.614039 25555 heartbeater.cc:507] Master 127.0.71.124:32895 requested a full tablet report, sending...
I20260929 01:55:02.614086 25104 ts_manager.cc:194] Registered new tserver with Master: d36e7a7040c34330b7346caf16813ae2 (127.0.71.66:45349)
I20260929 01:55:02.615058 25104 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.66@KRBTEST.COM'} at 127.0.71.66:37637
I20260929 01:55:02.615157 25181 ts_manager.cc:194] Registered new tserver with Master: d36e7a7040c34330b7346caf16813ae2 (127.0.71.66:45349)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:55:02.704986 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.67:0
--local_ip_for_outbound_sockets=127.0.71.67
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:35193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:55:03.013132 25415 heartbeater.cc:499] Master 127.0.71.125:35913 was elected leader, sending a full tablet report...
W20260929 01:55:03.068490 25569 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:55:03.068835 25569 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:55:03.069031 25569 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:55:03.077808 25569 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:55:03.078104 25569 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.67
I20260929 01:55:03.089995 25569 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.71.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:55:03.091558 25569 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:55:03.092933 25569 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:55:03.103495 25575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:03.103454 25574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:03.104506 25577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:03.104826 25569 server_base.cc:1061] running on GCE node
I20260929 01:55:03.105489 25569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:55:03.106472 25569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:55:03.107723 25569 hybrid_clock.cc:648] HybridClock initialized: now 1790646903107660 us; error 57 us; skew 500 ppm
I20260929 01:55:03.108091 25569 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:55:03 dist-test-slave-vz23 krb5kdc[24700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:55:03.111912 25569 init.cc:377] Logged in from keytab as kudu/127.0.71.67@KRBTEST.COM (short username kudu)
I20260929 01:55:03.114110 25569 webserver.cc:533] Webserver started at http://127.0.71.67:34477/ using document root <none> and password file <none>
I20260929 01:55:03.114949 25569 fs_manager.cc:362] Metadata directory not provided
I20260929 01:55:03.115044 25569 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:55:03.115337 25569 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:55:03.117805 25569 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/data/instance:
uuid: "37c0c95fb5c74d778c035456328b0558"
format_stamp: "Formatted at 2026-09-29 01:55:03 on dist-test-slave-vz23"
I20260929 01:55:03.118546 25569 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/wal/instance:
uuid: "37c0c95fb5c74d778c035456328b0558"
format_stamp: "Formatted at 2026-09-29 01:55:03 on dist-test-slave-vz23"
I20260929 01:55:03.124099 25569 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:55:03.127800 25584 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:03.129444 25569 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:55:03.129604 25569 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/wal
uuid: "37c0c95fb5c74d778c035456328b0558"
format_stamp: "Formatted at 2026-09-29 01:55:03 on dist-test-slave-vz23"
I20260929 01:55:03.129801 25569 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:55:03.153146 25569 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:55:03.157907 25569 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:55:03.158241 25569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:55:03.159241 25569 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:55:03.161208 25569 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:55:03.161283 25569 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:03.161417 25569 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:55:03.161466 25569 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:03.200086 25569 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.67:43143
I20260929 01:55:03.200184 25696 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.67:43143 every 8 connection(s)
I20260929 01:55:03.201687 25569 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
I20260929 01:55:03.211419 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 25569
I20260929 01:55:03.211747 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/wal/instance
Sep 29 01:55:03 dist-test-slave-vz23 krb5kdc[24700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:55:03 dist-test-slave-vz23 krb5kdc[24700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:55:03 dist-test-slave-vz23 krb5kdc[24700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:55:03.240612 25697 heartbeater.cc:344] Connected to a master server at 127.0.71.124:32895
I20260929 01:55:03.240747 25698 heartbeater.cc:344] Connected to a master server at 127.0.71.126:41391
I20260929 01:55:03.241118 25697 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:03.241611 25698 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:03.241647 25699 heartbeater.cc:344] Connected to a master server at 127.0.71.125:35913
I20260929 01:55:03.241825 25699 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:03.242928 25698 heartbeater.cc:507] Master 127.0.71.126:41391 requested a full tablet report, sending...
I20260929 01:55:03.243012 25697 heartbeater.cc:507] Master 127.0.71.124:32895 requested a full tablet report, sending...
I20260929 01:55:03.242978 25699 heartbeater.cc:507] Master 127.0.71.125:35913 requested a full tablet report, sending...
I20260929 01:55:03.245147 25027 ts_manager.cc:194] Registered new tserver with Master: 37c0c95fb5c74d778c035456328b0558 (127.0.71.67:43143)
I20260929 01:55:03.245152 25104 ts_manager.cc:194] Registered new tserver with Master: 37c0c95fb5c74d778c035456328b0558 (127.0.71.67:43143)
I20260929 01:55:03.245308 25181 ts_manager.cc:194] Registered new tserver with Master: 37c0c95fb5c74d778c035456328b0558 (127.0.71.67:43143)
I20260929 01:55:03.246269 25104 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.67@KRBTEST.COM'} at 127.0.71.67:57431
I20260929 01:55:03.260372 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:55:03.269618 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:55:03.269891 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 24996
I20260929 01:55:03.270952 25058 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:55:03.382328 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 24996
I20260929 01:55:03.403976 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:41391
--webserver_interface=127.0.71.126
--webserver_port=42299
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:35193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:55:03.421401 25090 consensus_peers.cc:597] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 -> Peer d6cf88e894de42e49d440d3b67403d37 (127.0.71.126:41391): Couldn't send request to peer d6cf88e894de42e49d440d3b67403d37. Status: Network error: Client connection negotiation failed: client connection to 127.0.71.126:41391: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260929 01:55:03.617514 25556 heartbeater.cc:646] Failed to heartbeat to 127.0.71.126:41391 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.71.126:41391: connect: Connection refused (error 111)
I20260929 01:55:03.617745 25557 heartbeater.cc:499] Master 127.0.71.125:35913 was elected leader, sending a full tablet report...
W20260929 01:55:03.764063 25708 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:55:03.764425 25708 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:55:03.764611 25708 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:55:03.773331 25708 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:55:03.773449 25708 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:55:03.773510 25708 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:55:03.773559 25708 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:55:03.784670 25708 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:41391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=42299
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:55:03.786295 25708 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:55:03.787631 25708 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:55:03.796502 25717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:03.796471 25716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:03.797099 25719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:03.797629 25708 server_base.cc:1061] running on GCE node
I20260929 01:55:03.798448 25708 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:55:03.799940 25708 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:55:03.801156 25708 hybrid_clock.cc:648] HybridClock initialized: now 1790646903801093 us; error 53 us; skew 500 ppm
I20260929 01:55:03.801541 25708 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:55:03 dist-test-slave-vz23 krb5kdc[24700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646903, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:55:03.805285 25708 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:55:03.807324 25708 webserver.cc:533] Webserver started at http://127.0.71.126:42299/ using document root <none> and password file <none>
I20260929 01:55:03.808107 25708 fs_manager.cc:362] Metadata directory not provided
I20260929 01:55:03.808266 25708 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:55:03.814971 25708 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.001s sys 0.004s
I20260929 01:55:03.819275 25726 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:03.820947 25708 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:55:03.821159 25708 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "d6cf88e894de42e49d440d3b67403d37"
format_stamp: "Formatted at 2026-09-29 01:54:59 on dist-test-slave-vz23"
I20260929 01:55:03.822033 25708 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:55:03.859827 25708 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:55:03.863476 25298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "38f49999d352471d8b1571385b4363c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:03.863778 25298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [sys.catalog]: This master's current role is: LEADER
I20260929 01:55:03.865593 25708 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:55:03.865899 25708 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:55:03.887058 25708 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:41391
I20260929 01:55:03.887094 25780 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:41391 every 8 connection(s)
I20260929 01:55:03.888824 25708 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:55:03.889552 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 25708
I20260929 01:55:03.889897 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 25284
I20260929 01:55:03.897020 25781 sys_catalog.cc:263] Verifying existing consensus state
Sep 29 01:55:03 dist-test-slave-vz23 krb5kdc[24700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 01:55:03.900532 25781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: Bootstrap starting.
I20260929 01:55:03.908108 25408 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:55:03.919275 25781 log.cc:826] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: Log is configured to *not* fsync() on all Append() calls
I20260929 01:55:03.929209 25781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:55:03.929767 25781 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: Bootstrap complete.
I20260929 01:55:03.937063 25781 raft_consensus.cc:359] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } }
I20260929 01:55:03.937803 25781 raft_consensus.cc:740] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6cf88e894de42e49d440d3b67403d37, State: Initialized, Role: FOLLOWER
I20260929 01:55:03.938666 25781 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } }
I20260929 01:55:03.939648 25787 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:03.939944 25787 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:55:03.940920 25781 sys_catalog.cc:565] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:55:03.952250 25798 catalog_manager.cc:1308] Loaded cluster ID: 311e41733bad4ff691115e2dbb636f4d
I20260929 01:55:03.952584 25798 catalog_manager.cc:1601] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: loading cluster ID for follower catalog manager: success
I20260929 01:55:03.958746 25798 catalog_manager.cc:1623] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: acquiring CA information for follower catalog manager: success
I20260929 01:55:03.963165 25798 catalog_manager.cc:1651] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:55:04.036507 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 25284
W20260929 01:55:04.055197 25166 connection.cc:573] server connection from 127.0.71.65:34575 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 01:55:04.055222 25089 connection.cc:573] server connection from 127.0.71.65:55267 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 01:55:04.056417 25801 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:55:04.035869 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:49007 (local address 127.0.71.126:41391)
0929 01:55:04.036067 (+ 198us) server_negotiation.cc:207] Beginning negotiation
0929 01:55:04.036074 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0929 01:55:04.055243 (+ 19169us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:49007: BlockingRecv error: recv got EOF from 127.0.71.65:49007 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 01:55:04.057462 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:38659
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--webserver_port=44163
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:35193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:55:04.248594 25699 heartbeater.cc:499] Master 127.0.71.125:35913 was elected leader, sending a full tablet report...
I20260929 01:55:04.260998 25746 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "37c0c95fb5c74d778c035456328b0558" instance_seqno: 1790646903191257) as {username='kudu', principal='kudu/127.0.71.67@KRBTEST.COM'} at 127.0.71.67:50747; Asking this server to re-register.
I20260929 01:55:04.262069 25698 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:04.264336 25746 ts_manager.cc:194] Registered new tserver with Master: 37c0c95fb5c74d778c035456328b0558 (127.0.71.67:43143)
I20260929 01:55:04.346850 25787 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38f49999d352471d8b1571385b4363c5. Latest consensus state: current_term: 1 leader_uuid: "38f49999d352471d8b1571385b4363c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:04.347141 25787 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:55:04.451258 25805 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:55:04.451627 25805 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:55:04.451788 25805 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:55:04.460775 25805 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:55:04.461033 25805 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:55:04.472681 25805 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:38659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=44163
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:55:04.474349 25805 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:55:04.475720 25805 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:55:04.486824 25814 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:04.486861 25812 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:04.486824 25811 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:04.487952 25805 server_base.cc:1061] running on GCE node
I20260929 01:55:04.488682 25805 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:55:04.489748 25805 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:55:04.490998 25805 hybrid_clock.cc:648] HybridClock initialized: now 1790646904490943 us; error 48 us; skew 500 ppm
I20260929 01:55:04.491350 25805 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:55:04 dist-test-slave-vz23 krb5kdc[24700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:55:04.494978 25805 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:55:04.496793 25805 webserver.cc:533] Webserver started at http://127.0.71.65:44163/ using document root <none> and password file <none>
I20260929 01:55:04.497560 25805 fs_manager.cc:362] Metadata directory not provided
I20260929 01:55:04.497716 25805 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:55:04.503669 25805 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260929 01:55:04.507426 25821 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:04.509145 25805 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 01:55:04.509392 25805 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "bc92f4e26ca24156a850ac6f0c03209c"
format_stamp: "Formatted at 2026-09-29 01:55:01 on dist-test-slave-vz23"
I20260929 01:55:04.510236 25805 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:55:04.535411 25805 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:55:04.544996 25805 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:55:04.545288 25805 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:55:04.546418 25805 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:55:04.548072 25805 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:55:04.548203 25805 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:04.548343 25805 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:55:04.548467 25805 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:04.589005 25805 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:38659
I20260929 01:55:04.589056 25933 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:38659 every 8 connection(s)
I20260929 01:55:04.590679 25805 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:55:04.591061 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 25805
Sep 29 01:55:04 dist-test-slave-vz23 krb5kdc[24700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646880, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for hive/127.0.0.1@KRBTEST.COM
Sep 29 01:55:04 dist-test-slave-vz23 krb5kdc[24700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:55:04 dist-test-slave-vz23 krb5kdc[24700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:55:04.629339 25936 heartbeater.cc:344] Connected to a master server at 127.0.71.125:35913
I20260929 01:55:04.629768 25936 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:04.630879 25936 heartbeater.cc:507] Master 127.0.71.125:35913 requested a full tablet report, sending...
I20260929 01:55:04.633041 25104 ts_manager.cc:194] Re-registered known tserver with Master: bc92f4e26ca24156a850ac6f0c03209c (127.0.71.65:38659)
I20260929 01:55:04.634094 25104 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:53723
I20260929 01:55:04.634441 25934 heartbeater.cc:344] Connected to a master server at 127.0.71.124:32895
I20260929 01:55:04.634601 25934 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:04.635278 25934 heartbeater.cc:507] Master 127.0.71.124:32895 requested a full tablet report, sending...
I20260929 01:55:04.636790 25181 ts_manager.cc:194] Re-registered known tserver with Master: bc92f4e26ca24156a850ac6f0c03209c (127.0.71.65:38659)
Sep 29 01:55:04 dist-test-slave-vz23 krb5kdc[24700](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646904, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:55:04.649857 25935 heartbeater.cc:344] Connected to a master server at 127.0.71.126:41391
I20260929 01:55:04.649902 25556 heartbeater.cc:344] Connected to a master server at 127.0.71.126:41391
I20260929 01:55:04.650064 25935 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:04.650513 25935 heartbeater.cc:507] Master 127.0.71.126:41391 requested a full tablet report, sending...
I20260929 01:55:04.651110 25746 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d36e7a7040c34330b7346caf16813ae2" instance_seqno: 1790646902551703) as {username='kudu', principal='kudu/127.0.71.66@KRBTEST.COM'} at 127.0.71.66:39131; Asking this server to re-register.
I20260929 01:55:04.651659 25556 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:04.651785 25745 ts_manager.cc:194] Registered new tserver with Master: bc92f4e26ca24156a850ac6f0c03209c (127.0.71.65:38659)
I20260929 01:55:04.652884 25745 ts_manager.cc:194] Registered new tserver with Master: d36e7a7040c34330b7346caf16813ae2 (127.0.71.66:45349)
I20260929 01:55:04.751850 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 25805
W20260929 01:55:04.753293 25949 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:55:04.749277 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:35302 (local address 127.0.71.65:38659)
0929 01:55:04.750076 (+ 799us) server_negotiation.cc:207] Beginning negotiation
0929 01:55:04.750083 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0929 01:55:04.750113 (+ 30us) server_negotiation.cc:408] Connection header received
0929 01:55:04.750180 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:55:04.750187 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:55:04.750417 (+ 230us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:55:04.750605 (+ 188us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:55:04.751720 (+ 1115us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35302: BlockingRecv error: recv got EOF from 127.0.0.1:35302 (error 108)
Metrics: {"server-negotiator.queue_time_us":661,"thread_start_us":411,"threads_started":1}
W20260929 01:55:04.754508 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 25805: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:38659: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:55:04.754684 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 25805
W20260929 01:55:04.754703 25950 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:55:04.752696 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:35310 (local address 127.0.71.65:38659)
0929 01:55:04.753325 (+ 629us) server_negotiation.cc:207] Beginning negotiation
0929 01:55:04.753331 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0929 01:55:04.753355 (+ 24us) server_negotiation.cc:408] Connection header received
0929 01:55:04.753416 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:55:04.753422 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:55:04.753729 (+ 307us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:55:04.753823 (+ 94us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:55:04.754397 (+ 574us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:35310: BlockingRecv error: recv got EOF from 127.0.0.1:35310 (error 108)
Metrics: {"server-negotiator.queue_time_us":509,"thread_start_us":368,"threads_started":1}
I20260929 01:55:04.771595 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:38659
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--webserver_port=44163
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:35193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:55:04.784008 25298 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "38f49999d352471d8b1571385b4363c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:04.784291 25298 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [sys.catalog]: This master's current role is: LEADER
W20260929 01:55:05.140642 25951 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:55:05.141057 25951 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:55:05.141242 25951 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:55:05.141350 25951 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:55:05.150970 25951 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:55:05.151281 25951 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:55:05.162635 25951 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:38659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=44163
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:55:05.164358 25951 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:55:05.165783 25951 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:55:05.176783 25960 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:05.176774 25959 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:05.177716 25962 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:05.178725 25951 server_base.cc:1061] running on GCE node
I20260929 01:55:05.179412 25951 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:55:05.180523 25951 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:55:05.181774 25951 hybrid_clock.cc:648] HybridClock initialized: now 1790646905181705 us; error 41 us; skew 500 ppm
I20260929 01:55:05.182121 25951 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:55:05 dist-test-slave-vz23 krb5kdc[24700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:55:05.185755 25951 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:55:05.187670 25951 webserver.cc:533] Webserver started at http://127.0.71.65:44163/ using document root <none> and password file <none>
I20260929 01:55:05.188431 25951 fs_manager.cc:362] Metadata directory not provided
I20260929 01:55:05.188593 25951 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:55:05.194319 25951 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:55:05.197482 25969 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:05.199023 25951 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:55:05.199211 25951 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "bc92f4e26ca24156a850ac6f0c03209c"
format_stamp: "Formatted at 2026-09-29 01:55:01 on dist-test-slave-vz23"
I20260929 01:55:05.199959 25951 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:55:05.226305 25951 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:55:05.231050 25951 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:55:05.231447 25951 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:55:05.232577 25951 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:55:05.234345 25951 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:55:05.234472 25951 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:05.234628 25951 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:55:05.234701 25951 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:55:05.235138 25951 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:55:05.235227 25951 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:55:05.235312 25951 fault_injection.cc:59] injecting fault for kudu (pid 25951): FLAGS_fault_before_start (process will exit)
I20260929 01:55:05.258919 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:38659
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--webserver_port=44163
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:35193
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:55:05.632239 25977 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:55:05.632757 25977 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:55:05.632948 25977 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:55:05.641894 25977 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:55:05.642223 25977 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:55:05.654774 25977 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35193
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:38659
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=44163
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:55:05.656679 25977 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:55:05.658136 25977 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:55:05.669035 25985 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:05.669024 25984 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:05.670387 25987 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:05.671839 25977 server_base.cc:1061] running on GCE node
I20260929 01:55:05.672636 25977 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:55:05.673974 25977 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:55:05.675280 25977 hybrid_clock.cc:648] HybridClock initialized: now 1790646905675204 us; error 58 us; skew 500 ppm
I20260929 01:55:05.675700 25977 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:55:05 dist-test-slave-vz23 krb5kdc[24700](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646905, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:55:05.684967 25977 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:55:05.686789 25977 webserver.cc:533] Webserver started at http://127.0.71.65:44163/ using document root <none> and password file <none>
I20260929 01:55:05.687547 25977 fs_manager.cc:362] Metadata directory not provided
I20260929 01:55:05.687654 25977 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:55:05.693578 25977 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:55:05.696619 25994 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:05.698102 25977 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:55:05.698289 25977 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "bc92f4e26ca24156a850ac6f0c03209c"
format_stamp: "Formatted at 2026-09-29 01:55:01 on dist-test-slave-vz23"
I20260929 01:55:05.699090 25977 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:55:05.712965 25977 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:55:05.717303 25977 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:55:05.717609 25977 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:55:05.718760 25977 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:55:05.720592 25977 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:55:05.720687 25977 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:05.720762 25977 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:55:05.720820 25977 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:05.765655 25977 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:38659
I20260929 01:55:05.765743 26106 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:38659 every 8 connection(s)
I20260929 01:55:05.767459 25977 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:55:05.775931 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 25977
I20260929 01:55:05.776319 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 25977
W20260929 01:55:05.793458 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 25977: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:38659: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:55:05.793596 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 25977
W20260929 01:55:05.810252 25982 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:55:05.771295 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:50875 (local address 127.0.71.126:41391)
0929 01:55:05.771562 (+ 267us) server_negotiation.cc:207] Beginning negotiation
0929 01:55:05.771572 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0929 01:55:05.787521 (+ 15949us) server_negotiation.cc:408] Connection header received
0929 01:55:05.788104 (+ 583us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:55:05.788114 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:55:05.788223 (+ 109us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:55:05.788312 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:55:05.790669 (+ 2357us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:55:05.792036 (+ 1367us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:55:05.810024 (+ 17988us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:50875: BlockingRecv error: recv got EOF from 127.0.71.65:50875 (error 108)
Metrics: {"server-negotiator.queue_time_us":94}
I20260929 01:55:05.810792 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 25427
W20260929 01:55:05.810989 26111 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:55:05.770963 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:57555 (local address 127.0.71.125:35913)
0929 01:55:05.787210 (+ 16247us) server_negotiation.cc:207] Beginning negotiation
0929 01:55:05.787215 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0929 01:55:05.787230 (+ 15us) server_negotiation.cc:408] Connection header received
0929 01:55:05.787282 (+ 52us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:55:05.787286 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:55:05.787345 (+ 59us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:55:05.787498 (+ 153us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:55:05.790669 (+ 3171us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:55:05.792004 (+ 1335us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:55:05.810018 (+ 18014us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:57555: BlockingRecv error: recv got EOF from 127.0.71.65:57555 (error 108)
Metrics: {"server-negotiator.queue_time_us":16154,"thread_start_us":16032,"threads_started":1}
W20260929 01:55:05.810990 26113 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:55:05.772919 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:53703 (local address 127.0.71.124:32895)
0929 01:55:05.785604 (+ 12685us) server_negotiation.cc:207] Beginning negotiation
0929 01:55:05.785614 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0929 01:55:05.785643 (+ 29us) server_negotiation.cc:408] Connection header received
0929 01:55:05.785730 (+ 87us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:55:05.785740 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:55:05.785871 (+ 131us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:55:05.786025 (+ 154us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:55:05.810016 (+ 23991us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:53703: BlockingRecv error: recv got EOF from 127.0.71.65:53703 (error 108)
Metrics: {"server-negotiator.queue_time_us":12546,"thread_start_us":12382,"threads_started":1}
W20260929 01:55:05.813581 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 25427: Not authorized: Client connection negotiation failed: client connection to 127.0.71.66:45349: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:55:05.813747 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 25427
I20260929 01:55:05.831050 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 25569
W20260929 01:55:05.834206 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 25569: Not authorized: Client connection negotiation failed: client connection to 127.0.71.67:43143: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:55:05.834345 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 25569
I20260929 01:55:05.849042 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 25708
W20260929 01:55:05.851577 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 25708: Not authorized: Client connection negotiation failed: client connection to 127.0.71.126:41391: server requires authentication, but client does not have Kerberos credentials available
W20260929 01:55:05.851728 25982 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:55:05.850123 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:60126 (local address 127.0.71.126:41391)
0929 01:55:05.850293 (+ 170us) server_negotiation.cc:207] Beginning negotiation
0929 01:55:05.850298 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0929 01:55:05.850344 (+ 46us) server_negotiation.cc:408] Connection header received
0929 01:55:05.850542 (+ 198us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:55:05.850547 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:55:05.850614 (+ 67us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:55:05.850693 (+ 79us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:55:05.851470 (+ 777us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60126: BlockingRecv error: recv got EOF from 127.0.0.1:60126 (error 108)
Metrics: {"server-negotiator.queue_time_us":63}
I20260929 01:55:05.851758 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 25708
I20260929 01:55:05.874068 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 25072
I20260929 01:55:05.875572 25134 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:55:05.991814 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 25072
I20260929 01:55:06.012964 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 25149
I20260929 01:55:06.014098 25211 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:55:06.149291 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 25149
Shutting down hive metastore.
2026-09-29T01:55:06Z chronyd exiting
Sep 29 01:55:06 dist-test-slave-vz23 krb5kdc[26127](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:55:06 dist-test-slave-vz23 krb5kdc[26127](info): set up 2 sockets
Sep 29 01:55:06 dist-test-slave-vz23 krb5kdc[26127](info): commencing operation
krb5kdc: starting...
W20260929 01:55:08.577437 285 mini_kdc.cc:121] Time spent starting KDC: real 2.014s user 0.002s sys 0.017s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-admin@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-user@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "joe-interloper@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 01:55:08 dist-test-slave-vz23 krb5kdc[26127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:55:08Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:55:08Z Disabled control of system clock
WARNING: no policy specified for hive/127.0.0.1@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "hive/127.0.0.1@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal hive/127.0.0.1 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/hive_127.0.0.1.keytab.
Entry for principal hive/127.0.0.1 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/hive_127.0.0.1.keytab.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5.conf
2026-09-29 01:55:09: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
Sep 29 01:55:12 dist-test-slave-vz23 krb5kdc[26127](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790646912, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 01:55:16 dist-test-slave-vz23 krb5kdc[26127](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790646916, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 01:55:16 dist-test-slave-vz23 krb5kdc[26127](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790646916, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.126@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.126@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:55:18.397337 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:41391
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:41709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:55:18.741416 26320 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:55:18.741818 26320 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:55:18.741971 26320 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:55:18.751324 26320 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:55:18.751477 26320 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:55:18.751613 26320 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:55:18.751703 26320 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:55:18.763025 26320 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:41709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:41391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:55:18.764799 26320 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:55:18.766235 26320 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:55:18.775010 26326 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:18.774950 26325 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:18.775863 26328 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:18.776418 26320 server_base.cc:1061] running on GCE node
I20260929 01:55:18.777278 26320 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:55:18.778757 26320 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:55:18.779965 26320 hybrid_clock.cc:648] HybridClock initialized: now 1790646918779908 us; error 57 us; skew 500 ppm
I20260929 01:55:18.780400 26320 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:55:18 dist-test-slave-vz23 krb5kdc[26127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:55:18.784126 26320 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:55:18.785967 26320 webserver.cc:533] Webserver started at http://127.0.71.126:43771/ using document root <none> and password file <none>
I20260929 01:55:18.786713 26320 fs_manager.cc:362] Metadata directory not provided
I20260929 01:55:18.786836 26320 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:55:18.792495 26320 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 01:55:18.795413 26335 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:18.796932 26320 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:55:18.797113 26320 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "d6cf88e894de42e49d440d3b67403d37"
format_stamp: "Formatted at 2026-09-29 01:54:59 on dist-test-slave-vz23"
I20260929 01:55:18.797923 26320 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:55:18.814363 26320 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:55:18.818771 26320 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:55:18.819069 26320 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:55:18.840555 26320 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:41391
I20260929 01:55:18.840574 26386 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:41391 every 8 connection(s)
I20260929 01:55:18.842289 26320 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:55:18.849574 26387 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:55:18.851930 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 26320
I20260929 01:55:18.852169 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:55:18.853282 26387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: Bootstrap starting.
I20260929 01:55:18.870170 26387 log.cc:826] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: Log is configured to *not* fsync() on all Append() calls
WARNING: no policy specified for kudu/127.0.71.125@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.125@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:55:18.879549 26387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:55:18.880108 26387 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: Bootstrap complete.
I20260929 01:55:18.887077 26387 raft_consensus.cc:359] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } }
I20260929 01:55:18.887852 26387 raft_consensus.cc:740] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6cf88e894de42e49d440d3b67403d37, State: Initialized, Role: FOLLOWER
I20260929 01:55:18.888700 26387 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } }
I20260929 01:55:18.889637 26392 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:18.889937 26392 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:55:18.890838 26387 sys_catalog.cc:565] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:55:18.901465 26403 catalog_manager.cc:1308] Loaded cluster ID: 311e41733bad4ff691115e2dbb636f4d
I20260929 01:55:18.901656 26403 catalog_manager.cc:1601] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: loading cluster ID for follower catalog manager: success
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.125 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.125 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:55:18.912914 26403 catalog_manager.cc:1623] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: acquiring CA information for follower catalog manager: success
I20260929 01:55:18.916848 26403 catalog_manager.cc:1651] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Sep 29 01:55:18 dist-test-slave-vz23 krb5kdc[26127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646918, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
WARNING: no policy specified for HTTP/127.0.71.125@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.125@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:55:18.938462 26320 master_runner.cc:187] Error getting master registration for 127.0.71.125:35913: Network error: Client connection negotiation failed: client connection to 127.0.71.125:35913: connect: Connection refused (error 111),
I20260929 01:55:18.942873 26320 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.125 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.125 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:55:18.968570 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:35913
--webserver_interface=127.0.71.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:41709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:55:19.350832 26409 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:55:19.351181 26409 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:55:19.351264 26409 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:55:19.359938 26409 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:55:19.360054 26409 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:55:19.360117 26409 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:55:19.360158 26409 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:55:19.371435 26409 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:41709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:35913
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:55:19.373256 26409 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:55:19.374642 26409 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:55:19.382970 26414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:19.383008 26415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:19.384469 26417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:19.384861 26409 server_base.cc:1061] running on GCE node
I20260929 01:55:19.385663 26409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:55:19.387159 26409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:55:19.388386 26409 hybrid_clock.cc:648] HybridClock initialized: now 1790646919388331 us; error 50 us; skew 500 ppm
I20260929 01:55:19.388847 26409 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:55:19 dist-test-slave-vz23 krb5kdc[26127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646919, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:55:19.392591 26409 init.cc:377] Logged in from keytab as kudu/127.0.71.125@KRBTEST.COM (short username kudu)
I20260929 01:55:19.394390 26409 webserver.cc:533] Webserver started at http://127.0.71.125:45111/ using document root <none> and password file <none>
I20260929 01:55:19.395190 26409 fs_manager.cc:362] Metadata directory not provided
I20260929 01:55:19.395308 26409 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:55:19.401093 26409 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:55:19.404357 26424 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:19.405864 26409 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:55:19.406014 26409 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "38f49999d352471d8b1571385b4363c5"
format_stamp: "Formatted at 2026-09-29 01:54:59 on dist-test-slave-vz23"
I20260929 01:55:19.406757 26409 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:55:19.450637 26409 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:55:19.454856 26409 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:55:19.455090 26409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:55:19.476347 26409 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:35913
I20260929 01:55:19.476369 26475 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:35913 every 8 connection(s)
I20260929 01:55:19.478050 26409 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:55:19.480230 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 26409
I20260929 01:55:19.480419 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:55:19.487743 26476 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:55:19.492256 26476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5: Bootstrap starting.
WARNING: no policy specified for kudu/127.0.71.124@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.124@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:55:19.510987 26476 log.cc:826] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5: Log is configured to *not* fsync() on all Append() calls
I20260929 01:55:19.520169 26476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:55:19.520794 26476 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5: Bootstrap complete.
I20260929 01:55:19.527755 26476 raft_consensus.cc:359] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } }
I20260929 01:55:19.528625 26476 raft_consensus.cc:740] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 38f49999d352471d8b1571385b4363c5, State: Initialized, Role: FOLLOWER
I20260929 01:55:19.529495 26476 consensus_queue.cc:260] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } }
I20260929 01:55:19.530468 26481 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:19.530853 26481 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:55:19.531662 26476 sys_catalog.cc:565] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [sys.catalog]: configured and running, proceeding with master startup.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.124 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.124 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:55:19.542975 26492 catalog_manager.cc:1308] Loaded cluster ID: 311e41733bad4ff691115e2dbb636f4d
I20260929 01:55:19.543175 26492 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5: loading cluster ID for follower catalog manager: success
I20260929 01:55:19.548358 26492 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5: acquiring CA information for follower catalog manager: success
I20260929 01:55:19.551824 26492 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for HTTP/127.0.71.124@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.124@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:55:19.577780 26409 master_runner.cc:187] Error getting master registration for 127.0.71.124:32895: Network error: Client connection negotiation failed: client connection to 127.0.71.124:32895: connect: Connection refused (error 111),
I20260929 01:55:19.581976 26409 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.124 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.124 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:55:19.600134 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:32895
--webserver_interface=127.0.71.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:41709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:55:19.979030 26499 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:55:19.979389 26499 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:55:19.979524 26499 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:55:19.989559 26499 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:55:19.989684 26499 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:55:19.989820 26499 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:55:19.989912 26499 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:55:20.002377 26499 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:41709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:32895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:55:20.004086 26499 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:55:20.005431 26499 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:55:20.014382 26504 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:20.014354 26505 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:20.014770 26507 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:20.015442 26499 server_base.cc:1061] running on GCE node
I20260929 01:55:20.016223 26499 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:55:20.017778 26499 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:55:20.019007 26499 hybrid_clock.cc:648] HybridClock initialized: now 1790646920018953 us; error 55 us; skew 500 ppm
I20260929 01:55:20.019399 26499 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:55:20 dist-test-slave-vz23 krb5kdc[26127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:55:20.023026 26499 init.cc:377] Logged in from keytab as kudu/127.0.71.124@KRBTEST.COM (short username kudu)
I20260929 01:55:20.024765 26499 webserver.cc:533] Webserver started at http://127.0.71.124:33051/ using document root <none> and password file <none>
I20260929 01:55:20.025519 26499 fs_manager.cc:362] Metadata directory not provided
I20260929 01:55:20.025676 26499 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:55:20.031386 26499 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.005s
I20260929 01:55:20.034750 26514 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:20.036330 26499 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:55:20.036556 26499 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "978f92c285a44324bb56bc9bf17fbce0"
format_stamp: "Formatted at 2026-09-29 01:55:00 on dist-test-slave-vz23"
I20260929 01:55:20.037349 26499 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:55:20.063383 26499 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:55:20.068011 26499 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:55:20.068389 26499 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:55:20.091792 26499 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:32895
I20260929 01:55:20.091965 26565 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:32895 every 8 connection(s)
I20260929 01:55:20.093772 26499 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:55:20.099511 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 26499
I20260929 01:55:20.099772 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:55:20.101969 26566 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:55:20.108235 26566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0: Bootstrap starting.
Sep 29 01:55:20 dist-test-slave-vz23 krb5kdc[26127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:55:20.130873 26566 log.cc:826] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0: Log is configured to *not* fsync() on all Append() calls
Sep 29 01:55:20 dist-test-slave-vz23 krb5kdc[26127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:55:20.142604 26566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:55:20.143244 26566 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0: Bootstrap complete.
Sep 29 01:55:20 dist-test-slave-vz23 krb5kdc[26127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:55:20.152151 26566 raft_consensus.cc:359] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } }
I20260929 01:55:20.153049 26566 raft_consensus.cc:740] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 978f92c285a44324bb56bc9bf17fbce0, State: Initialized, Role: FOLLOWER
I20260929 01:55:20.153898 26566 consensus_queue.cc:260] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } }
I20260929 01:55:20.154837 26573 sys_catalog.cc:455] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:20.155221 26573 sys_catalog.cc:458] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:55:20.156104 26566 sys_catalog.cc:565] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:55:20.167546 26584 catalog_manager.cc:1308] Loaded cluster ID: 311e41733bad4ff691115e2dbb636f4d
I20260929 01:55:20.167670 26584 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0: loading cluster ID for follower catalog manager: success
I20260929 01:55:20.172479 26584 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0: acquiring CA information for follower catalog manager: success
Sep 29 01:55:20 dist-test-slave-vz23 krb5kdc[26127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:55:20.174853 26584 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:55:20.204310 26499 master_runner.cc:264] No leader master found from master 978f92c285a44324bb56bc9bf17fbce0
I20260929 01:55:20.204504 26499 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:55:20.208830 26587 raft_consensus.cc:493] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:55:20.209180 26587 raft_consensus.cc:515] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } }
I20260929 01:55:20.211797 26587 leader_election.cc:290] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 38f49999d352471d8b1571385b4363c5 (127.0.71.125:35913), 978f92c285a44324bb56bc9bf17fbce0 (127.0.71.124:32895)
I20260929 01:55:20.220080 26541 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6cf88e894de42e49d440d3b67403d37" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "978f92c285a44324bb56bc9bf17fbce0" is_pre_election: true
I20260929 01:55:20.220304 26451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6cf88e894de42e49d440d3b67403d37" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38f49999d352471d8b1571385b4363c5" is_pre_election: true
I20260929 01:55:20.220799 26451 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6cf88e894de42e49d440d3b67403d37 in term 1.
I20260929 01:55:20.220808 26541 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d6cf88e894de42e49d440d3b67403d37 in term 1.
I20260929 01:55:20.221498 26337 leader_election.cc:304] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38f49999d352471d8b1571385b4363c5, d6cf88e894de42e49d440d3b67403d37; no voters:
I20260929 01:55:20.221896 26587 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 01:55:20.222043 26587 raft_consensus.cc:493] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:55:20.222150 26587 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:55:20.226097 26587 raft_consensus.cc:515] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } }
I20260929 01:55:20.227085 26587 leader_election.cc:290] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [CANDIDATE]: Term 2 election: Requested vote from peers 38f49999d352471d8b1571385b4363c5 (127.0.71.125:35913), 978f92c285a44324bb56bc9bf17fbce0 (127.0.71.124:32895)
I20260929 01:55:20.227466 26451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6cf88e894de42e49d440d3b67403d37" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "38f49999d352471d8b1571385b4363c5"
I20260929 01:55:20.227517 26541 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d6cf88e894de42e49d440d3b67403d37" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "978f92c285a44324bb56bc9bf17fbce0"
I20260929 01:55:20.227691 26451 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:55:20.227717 26541 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:55:20.231295 26451 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6cf88e894de42e49d440d3b67403d37 in term 2.
I20260929 01:55:20.231751 26337 leader_election.cc:304] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 38f49999d352471d8b1571385b4363c5, d6cf88e894de42e49d440d3b67403d37; no voters:
I20260929 01:55:20.232102 26587 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:55:20.232659 26587 raft_consensus.cc:697] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [term 2 LEADER]: Becoming Leader. State: Replica: d6cf88e894de42e49d440d3b67403d37, State: Running, Role: LEADER
I20260929 01:55:20.233177 26587 consensus_queue.cc:237] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } }
I20260929 01:55:20.233649 26541 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d6cf88e894de42e49d440d3b67403d37 in term 2.
I20260929 01:55:20.235896 26590 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6cf88e894de42e49d440d3b67403d37. Latest consensus state: current_term: 2 leader_uuid: "d6cf88e894de42e49d440d3b67403d37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:20.236287 26590 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [sys.catalog]: This master's current role is: LEADER
I20260929 01:55:20.237088 26592 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:55:20.239563 26592 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:55:20.240752 26592 catalog_manager.cc:1308] Loaded cluster ID: 311e41733bad4ff691115e2dbb636f4d
I20260929 01:55:20.240846 26592 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:55:20.242136 26592 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:55:20.242950 26592 catalog_manager.cc:6230] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: Loaded TSK: 0
I20260929 01:55:20.243826 26592 catalog_manager.cc:1563] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.65@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.65@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:55:20.356336 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:41709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:55:20.704365 26541 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [term 2 FOLLOWER]: Refusing update from remote peer d6cf88e894de42e49d440d3b67403d37: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 01:55:20.705554 26590 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:55:20.708319 26601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6cf88e894de42e49d440d3b67403d37. Latest consensus state: current_term: 2 leader_uuid: "d6cf88e894de42e49d440d3b67403d37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:20.708765 26601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:55:20.710719 26590 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "d6cf88e894de42e49d440d3b67403d37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:20.711009 26590 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [sys.catalog]: This master's current role is: LEADER
I20260929 01:55:20.711828 26451 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [term 2 FOLLOWER]: Refusing update from remote peer d6cf88e894de42e49d440d3b67403d37: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 01:55:20.712798 26590 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [LEADER]: Connected to new peer: Peer: permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260929 01:55:20.713634 26597 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:55:20.714162 26597 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:55:20.714399 26597 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260929 01:55:20.725333 26601 sys_catalog.cc:455] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "d6cf88e894de42e49d440d3b67403d37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:20.725688 26601 sys_catalog.cc:458] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:55:20.727998 26609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "d6cf88e894de42e49d440d3b67403d37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:20.728323 26609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:55:20.728788 26606 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d6cf88e894de42e49d440d3b67403d37. Latest consensus state: current_term: 2 leader_uuid: "d6cf88e894de42e49d440d3b67403d37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:20.729091 26606 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:55:20.730403 26597 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:55:20.730638 26597 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:55:20.742164 26597 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:41709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:55:20.743854 26597 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:55:20.745353 26597 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:55:20.755555 26611 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:20.755602 26612 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:20.756690 26614 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:20.757011 26597 server_base.cc:1061] running on GCE node
I20260929 01:55:20.757668 26597 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:55:20.758661 26597 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:55:20.759928 26597 hybrid_clock.cc:648] HybridClock initialized: now 1790646920759881 us; error 41 us; skew 500 ppm
I20260929 01:55:20.760306 26597 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:55:20 dist-test-slave-vz23 krb5kdc[26127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:55:20.764015 26597 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:55:20.765964 26597 webserver.cc:533] Webserver started at http://127.0.71.65:35583/ using document root <none> and password file <none>
I20260929 01:55:20.766698 26597 fs_manager.cc:362] Metadata directory not provided
I20260929 01:55:20.766835 26597 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:55:20.772326 26597 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:55:20.775111 26621 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:20.776772 26597 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:55:20.777009 26597 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "bc92f4e26ca24156a850ac6f0c03209c"
format_stamp: "Formatted at 2026-09-29 01:55:01 on dist-test-slave-vz23"
I20260929 01:55:20.777818 26597 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:55:20.809120 26597 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:55:20.813858 26597 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:55:20.814163 26597 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:55:20.815294 26597 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:55:20.817076 26597 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:55:20.817174 26597 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:20.817324 26597 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:55:20.817410 26597 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:20.854882 26597 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:36101
I20260929 01:55:20.855001 26733 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:36101 every 8 connection(s)
I20260929 01:55:20.856523 26597 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:55:20.861830 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 26597
I20260929 01:55:20.862072 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
Sep 29 01:55:20 dist-test-slave-vz23 krb5kdc[26127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:55:20 dist-test-slave-vz23 krb5kdc[26127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:55:20 dist-test-slave-vz23 krb5kdc[26127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646920, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.66@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.66@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:55:20.911741 26736 heartbeater.cc:344] Connected to a master server at 127.0.71.125:35913
I20260929 01:55:20.912245 26736 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:20.913038 26734 heartbeater.cc:344] Connected to a master server at 127.0.71.124:32895
I20260929 01:55:20.913177 26734 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:20.913399 26736 heartbeater.cc:507] Master 127.0.71.125:35913 requested a full tablet report, sending...
I20260929 01:55:20.913729 26734 heartbeater.cc:507] Master 127.0.71.124:32895 requested a full tablet report, sending...
I20260929 01:55:20.914294 26735 heartbeater.cc:344] Connected to a master server at 127.0.71.126:41391
I20260929 01:55:20.914436 26735 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:20.914907 26735 heartbeater.cc:507] Master 127.0.71.126:41391 requested a full tablet report, sending...
I20260929 01:55:20.916193 26531 ts_manager.cc:194] Registered new tserver with Master: bc92f4e26ca24156a850ac6f0c03209c (127.0.71.65:36101)
I20260929 01:55:20.916685 26352 ts_manager.cc:194] Registered new tserver with Master: bc92f4e26ca24156a850ac6f0c03209c (127.0.71.65:36101)
I20260929 01:55:20.916757 26441 ts_manager.cc:194] Registered new tserver with Master: bc92f4e26ca24156a850ac6f0c03209c (127.0.71.65:36101)
I20260929 01:55:20.918612 26352 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:48987
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.66 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.66 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.66@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.66@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.66 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.66 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:55:20.988111 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.66:0
--local_ip_for_outbound_sockets=127.0.71.66
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:41709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:55:21.167017 26590 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "d6cf88e894de42e49d440d3b67403d37" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:21.167321 26590 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [sys.catalog]: This master's current role is: LEADER
W20260929 01:55:21.345057 26748 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:55:21.345368 26748 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:55:21.345476 26748 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:55:21.353861 26748 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:55:21.354066 26748 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.66
I20260929 01:55:21.365265 26748 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:41709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.71.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:55:21.366859 26748 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:55:21.368130 26748 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:55:21.378479 26753 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:21.378440 26754 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:21.378403 26756 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:21.379311 26748 server_base.cc:1061] running on GCE node
I20260929 01:55:21.379905 26748 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:55:21.380941 26748 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:55:21.382253 26748 hybrid_clock.cc:648] HybridClock initialized: now 1790646921382175 us; error 59 us; skew 500 ppm
I20260929 01:55:21.382689 26748 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:55:21 dist-test-slave-vz23 krb5kdc[26127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:55:21.386833 26748 init.cc:377] Logged in from keytab as kudu/127.0.71.66@KRBTEST.COM (short username kudu)
I20260929 01:55:21.388935 26748 webserver.cc:533] Webserver started at http://127.0.71.66:32809/ using document root <none> and password file <none>
I20260929 01:55:21.389745 26748 fs_manager.cc:362] Metadata directory not provided
I20260929 01:55:21.389900 26748 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:55:21.395732 26748 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 01:55:21.398622 26763 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:21.400054 26748 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:55:21.400238 26748 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/wal
uuid: "d36e7a7040c34330b7346caf16813ae2"
format_stamp: "Formatted at 2026-09-29 01:55:02 on dist-test-slave-vz23"
I20260929 01:55:21.401048 26748 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:55:21.416882 26748 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:55:21.428761 26748 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:55:21.429051 26748 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:55:21.430079 26748 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:55:21.431697 26748 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:55:21.431843 26748 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:21.431972 26748 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:55:21.432060 26748 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:21.468055 26748 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.66:40035
I20260929 01:55:21.468142 26875 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.66:40035 every 8 connection(s)
I20260929 01:55:21.469635 26748 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
I20260929 01:55:21.474990 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 26748
I20260929 01:55:21.475245 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-1/wal/instance
Sep 29 01:55:21 dist-test-slave-vz23 krb5kdc[26127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:55:21 dist-test-slave-vz23 krb5kdc[26127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:55:21 dist-test-slave-vz23 krb5kdc[26127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:55:21.528050 26877 heartbeater.cc:344] Connected to a master server at 127.0.71.126:41391
I20260929 01:55:21.528081 26878 heartbeater.cc:344] Connected to a master server at 127.0.71.125:35913
I20260929 01:55:21.528163 26876 heartbeater.cc:344] Connected to a master server at 127.0.71.124:32895
I20260929 01:55:21.528635 26878 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:21.528636 26877 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:21.528635 26876 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:21.529911 26877 heartbeater.cc:507] Master 127.0.71.126:41391 requested a full tablet report, sending...
I20260929 01:55:21.529918 26878 heartbeater.cc:507] Master 127.0.71.125:35913 requested a full tablet report, sending...
I20260929 01:55:21.530007 26876 heartbeater.cc:507] Master 127.0.71.124:32895 requested a full tablet report, sending...
I20260929 01:55:21.531981 26441 ts_manager.cc:194] Registered new tserver with Master: d36e7a7040c34330b7346caf16813ae2 (127.0.71.66:40035)
I20260929 01:55:21.532089 26531 ts_manager.cc:194] Registered new tserver with Master: d36e7a7040c34330b7346caf16813ae2 (127.0.71.66:40035)
I20260929 01:55:21.533174 26352 ts_manager.cc:194] Registered new tserver with Master: d36e7a7040c34330b7346caf16813ae2 (127.0.71.66:40035)
I20260929 01:55:21.534013 26352 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.66@KRBTEST.COM'} at 127.0.71.66:41019
WARNING: no policy specified for kudu/127.0.71.67@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.67@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.67 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.67 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.67@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.67@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.67 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.67 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:55:21.609709 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.67:0
--local_ip_for_outbound_sockets=127.0.71.67
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:41709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:55:21.921738 26735 heartbeater.cc:499] Master 127.0.71.126:41391 was elected leader, sending a full tablet report...
W20260929 01:55:21.943959 26890 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:55:21.944345 26890 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:55:21.944551 26890 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:55:21.953078 26890 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:55:21.953326 26890 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.67
I20260929 01:55:21.964257 26890 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:41709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.71.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:55:21.965973 26890 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:55:21.967454 26890 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:55:21.978825 26895 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:21.978986 26896 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:21.979025 26898 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:21.979910 26890 server_base.cc:1061] running on GCE node
I20260929 01:55:21.980620 26890 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:55:21.981623 26890 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:55:21.982852 26890 hybrid_clock.cc:648] HybridClock initialized: now 1790646921982780 us; error 60 us; skew 500 ppm
I20260929 01:55:21.983184 26890 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:55:21 dist-test-slave-vz23 krb5kdc[26127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:55:21.987205 26890 init.cc:377] Logged in from keytab as kudu/127.0.71.67@KRBTEST.COM (short username kudu)
I20260929 01:55:21.989027 26890 webserver.cc:533] Webserver started at http://127.0.71.67:33959/ using document root <none> and password file <none>
I20260929 01:55:21.989727 26890 fs_manager.cc:362] Metadata directory not provided
I20260929 01:55:21.989888 26890 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:55:21.995651 26890 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:55:21.998628 26905 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:22.000089 26890 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:55:22.000276 26890 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/wal
uuid: "37c0c95fb5c74d778c035456328b0558"
format_stamp: "Formatted at 2026-09-29 01:55:03 on dist-test-slave-vz23"
I20260929 01:55:22.001082 26890 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:55:22.025995 26890 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:55:22.030367 26890 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:55:22.030661 26890 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:55:22.031806 26890 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:55:22.033597 26890 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:55:22.033691 26890 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:22.033769 26890 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:55:22.033818 26890 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:22.069262 26890 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.67:34769
I20260929 01:55:22.069308 27017 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.67:34769 every 8 connection(s)
I20260929 01:55:22.070811 26890 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
I20260929 01:55:22.073400 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 26890
I20260929 01:55:22.073616 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-2/wal/instance
Sep 29 01:55:22 dist-test-slave-vz23 krb5kdc[26127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:55:22 dist-test-slave-vz23 krb5kdc[26127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:55:22 dist-test-slave-vz23 krb5kdc[26127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646921, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:55:22.103449 27020 heartbeater.cc:344] Connected to a master server at 127.0.71.125:35913
I20260929 01:55:22.103547 27018 heartbeater.cc:344] Connected to a master server at 127.0.71.124:32895
I20260929 01:55:22.103861 27020 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:22.103873 27018 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:22.104812 27018 heartbeater.cc:507] Master 127.0.71.124:32895 requested a full tablet report, sending...
I20260929 01:55:22.106185 27020 heartbeater.cc:507] Master 127.0.71.125:35913 requested a full tablet report, sending...
I20260929 01:55:22.106983 26531 ts_manager.cc:194] Registered new tserver with Master: 37c0c95fb5c74d778c035456328b0558 (127.0.71.67:34769)
I20260929 01:55:22.107544 26441 ts_manager.cc:194] Registered new tserver with Master: 37c0c95fb5c74d778c035456328b0558 (127.0.71.67:34769)
I20260929 01:55:22.110999 27019 heartbeater.cc:344] Connected to a master server at 127.0.71.126:41391
I20260929 01:55:22.111150 27019 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:22.111667 27019 heartbeater.cc:507] Master 127.0.71.126:41391 requested a full tablet report, sending...
I20260929 01:55:22.112941 26351 ts_manager.cc:194] Registered new tserver with Master: 37c0c95fb5c74d778c035456328b0558 (127.0.71.67:34769)
I20260929 01:55:22.114338 26351 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.67@KRBTEST.COM'} at 127.0.71.67:45993
I20260929 01:55:22.117384 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:55:22.125065 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:55:22.125312 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 26320
I20260929 01:55:22.126179 26382 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:55:22.237512 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 26320
I20260929 01:55:22.259902 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:41391
--webserver_interface=127.0.71.126
--webserver_port=43771
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:41709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:55:22.536698 26877 heartbeater.cc:499] Master 127.0.71.126:41391 was elected leader, sending a full tablet report...
W20260929 01:55:22.538332 26877 heartbeater.cc:646] Failed to heartbeat to 127.0.71.126:41391 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.71.126:41391: connect: Connection refused (error 111)
W20260929 01:55:22.603488 27029 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:55:22.603811 27029 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:55:22.603981 27029 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:55:22.612906 27029 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:55:22.613009 27029 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:55:22.613116 27029 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:55:22.613201 27029 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:55:22.625244 27029 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:41709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:41391
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=43771
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:55:22.627064 27029 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:55:22.628707 27029 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:55:22.637982 27036 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:22.637879 27035 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:22.638392 27038 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:22.639257 27029 server_base.cc:1061] running on GCE node
I20260929 01:55:22.640246 27029 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:55:22.641912 27029 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:55:22.643123 27029 hybrid_clock.cc:648] HybridClock initialized: now 1790646922643080 us; error 34 us; skew 500 ppm
I20260929 01:55:22.643576 27029 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:55:22 dist-test-slave-vz23 krb5kdc[26127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646922, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:55:22.647668 27029 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:55:22.649657 27029 webserver.cc:533] Webserver started at http://127.0.71.126:43771/ using document root <none> and password file <none>
I20260929 01:55:22.650478 27029 fs_manager.cc:362] Metadata directory not provided
I20260929 01:55:22.650585 27029 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:55:22.656945 27029 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:55:22.660274 27045 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:22.661790 27029 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:55:22.661993 27029 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "d6cf88e894de42e49d440d3b67403d37"
format_stamp: "Formatted at 2026-09-29 01:54:59 on dist-test-slave-vz23"
I20260929 01:55:22.662804 27029 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:55:22.699452 27029 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:55:22.703717 27029 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:55:22.704015 27029 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:55:22.727025 27029 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:41391
I20260929 01:55:22.727051 27096 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:41391 every 8 connection(s)
I20260929 01:55:22.728842 27029 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:55:22.734656 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 27029
I20260929 01:55:22.735057 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 26597
I20260929 01:55:22.737615 27097 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:55:22.742910 27097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: Bootstrap starting.
Sep 29 01:55:22 dist-test-slave-vz23 krb5kdc[26127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 01:55:22.757134 26729 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:55:22.766718 27097 log.cc:826] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: Log is configured to *not* fsync() on all Append() calls
I20260929 01:55:22.779599 27097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:55:22.780184 27097 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: Bootstrap complete.
I20260929 01:55:22.788187 27097 raft_consensus.cc:359] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } }
I20260929 01:55:22.789090 27097 raft_consensus.cc:740] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: d6cf88e894de42e49d440d3b67403d37, State: Initialized, Role: FOLLOWER
I20260929 01:55:22.789968 27097 consensus_queue.cc:260] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } }
I20260929 01:55:22.791090 27103 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:22.791445 27103 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:55:22.792783 27097 sys_catalog.cc:565] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:55:22.804492 27114 catalog_manager.cc:1308] Loaded cluster ID: 311e41733bad4ff691115e2dbb636f4d
I20260929 01:55:22.805007 27114 catalog_manager.cc:1601] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: loading cluster ID for follower catalog manager: success
I20260929 01:55:22.809866 27114 catalog_manager.cc:1623] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: acquiring CA information for follower catalog manager: success
I20260929 01:55:22.816349 27114 catalog_manager.cc:1651] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:55:22.851841 27029 master_runner.cc:264] No leader master found from master d6cf88e894de42e49d440d3b67403d37
I20260929 01:55:22.852015 27029 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:55:22.878271 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 26597
I20260929 01:55:22.895529 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:36101
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--webserver_port=35583
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:41709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:55:23.117280 27019 heartbeater.cc:499] Master 127.0.71.126:41391 was elected leader, sending a full tablet report...
I20260929 01:55:23.128683 27062 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "37c0c95fb5c74d778c035456328b0558" instance_seqno: 1790646922060756) as {username='kudu', principal='kudu/127.0.71.67@KRBTEST.COM'} at 127.0.71.67:60249; Asking this server to re-register.
I20260929 01:55:23.129698 27019 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:23.131541 27062 ts_manager.cc:194] Registered new tserver with Master: 37c0c95fb5c74d778c035456328b0558 (127.0.71.67:34769)
I20260929 01:55:23.252167 27122 raft_consensus.cc:493] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d6cf88e894de42e49d440d3b67403d37)
I20260929 01:55:23.252424 27122 raft_consensus.cc:515] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } }
I20260929 01:55:23.254323 27122 leader_election.cc:290] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d6cf88e894de42e49d440d3b67403d37 (127.0.71.126:41391), 978f92c285a44324bb56bc9bf17fbce0 (127.0.71.124:32895)
I20260929 01:55:23.263818 26541 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38f49999d352471d8b1571385b4363c5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "978f92c285a44324bb56bc9bf17fbce0" is_pre_election: true
I20260929 01:55:23.264956 27072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38f49999d352471d8b1571385b4363c5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d6cf88e894de42e49d440d3b67403d37" is_pre_election: true
I20260929 01:55:23.265604 27072 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 38f49999d352471d8b1571385b4363c5 in term 2.
I20260929 01:55:23.266185 26427 leader_election.cc:304] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38f49999d352471d8b1571385b4363c5, d6cf88e894de42e49d440d3b67403d37; no voters: 978f92c285a44324bb56bc9bf17fbce0
I20260929 01:55:23.266594 27122 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 01:55:23.266738 27122 raft_consensus.cc:493] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [term 2 FOLLOWER]: Starting leader election (detected failure of leader d6cf88e894de42e49d440d3b67403d37)
I20260929 01:55:23.266845 27122 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:55:23.270119 27122 raft_consensus.cc:515] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } }
I20260929 01:55:23.270880 27122 leader_election.cc:290] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [CANDIDATE]: Term 3 election: Requested vote from peers d6cf88e894de42e49d440d3b67403d37 (127.0.71.126:41391), 978f92c285a44324bb56bc9bf17fbce0 (127.0.71.124:32895)
I20260929 01:55:23.271459 27072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38f49999d352471d8b1571385b4363c5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d6cf88e894de42e49d440d3b67403d37"
I20260929 01:55:23.271461 26541 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "38f49999d352471d8b1571385b4363c5" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "978f92c285a44324bb56bc9bf17fbce0"
W20260929 01:55:23.271412 27120 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
I20260929 01:55:23.271672 27072 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [term 2 FOLLOWER]: Advancing to term 3
W20260929 01:55:23.271844 27120 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:55:23.272032 27120 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260929 01:55:23.277346 27072 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 38f49999d352471d8b1571385b4363c5 in term 3.
I20260929 01:55:23.277906 26427 leader_election.cc:304] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 38f49999d352471d8b1571385b4363c5, d6cf88e894de42e49d440d3b67403d37; no voters: 978f92c285a44324bb56bc9bf17fbce0
I20260929 01:55:23.278326 27122 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 01:55:23.278846 27122 raft_consensus.cc:697] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [term 3 LEADER]: Becoming Leader. State: Replica: 38f49999d352471d8b1571385b4363c5, State: Running, Role: LEADER
I20260929 01:55:23.279376 27122 consensus_queue.cc:237] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } }
I20260929 01:55:23.281486 27128 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38f49999d352471d8b1571385b4363c5. Latest consensus state: current_term: 3 leader_uuid: "38f49999d352471d8b1571385b4363c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:23.281839 27128 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [sys.catalog]: This master's current role is: LEADER
I20260929 01:55:23.282581 27130 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:55:23.285076 27130 catalog_manager.cc:1533] Initializing Kudu cluster ID...
W20260929 01:55:23.285475 27120 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:55:23.285774 27120 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:55:23.286103 27130 catalog_manager.cc:1308] Loaded cluster ID: 311e41733bad4ff691115e2dbb636f4d
I20260929 01:55:23.286186 27130 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:55:23.287566 27130 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:55:23.288690 27130 catalog_manager.cc:6230] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5: Loaded TSK: 0
I20260929 01:55:23.289796 27130 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:55:23.299604 27120 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:41709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:36101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=35583
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:55:23.301291 27120 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:55:23.302700 27120 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:55:23.313360 27132 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:23.313385 27133 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:23.314029 27135 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:23.314224 27120 server_base.cc:1061] running on GCE node
I20260929 01:55:23.314805 27120 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:55:23.315759 27120 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:55:23.316967 27120 hybrid_clock.cc:648] HybridClock initialized: now 1790646923316911 us; error 50 us; skew 500 ppm
I20260929 01:55:23.317313 27120 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:55:23 dist-test-slave-vz23 krb5kdc[26127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:55:23.321288 27120 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:55:23.323135 27120 webserver.cc:533] Webserver started at http://127.0.71.65:35583/ using document root <none> and password file <none>
I20260929 01:55:23.323907 27120 fs_manager.cc:362] Metadata directory not provided
I20260929 01:55:23.324051 27120 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:55:23.330045 27120 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:55:23.333161 27142 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:23.334695 27120 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:55:23.334928 27120 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "bc92f4e26ca24156a850ac6f0c03209c"
format_stamp: "Formatted at 2026-09-29 01:55:01 on dist-test-slave-vz23"
I20260929 01:55:23.335741 27120 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:55:23.353514 27120 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:55:23.357831 27120 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:55:23.358117 27120 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:55:23.359161 27120 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:55:23.360936 27120 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:55:23.361066 27120 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:23.361196 27120 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:55:23.361287 27120 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:23.398089 27120 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:36101
I20260929 01:55:23.398134 27254 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:36101 every 8 connection(s)
I20260929 01:55:23.399682 27120 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:55:23.404382 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 27120
Sep 29 01:55:23 dist-test-slave-vz23 krb5kdc[26127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646908, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for hive/127.0.0.1@KRBTEST.COM
Sep 29 01:55:23 dist-test-slave-vz23 krb5kdc[26127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:55:23 dist-test-slave-vz23 krb5kdc[26127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:55:23 dist-test-slave-vz23 krb5kdc[26127](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:55:23.435160 27257 heartbeater.cc:344] Connected to a master server at 127.0.71.125:35913
I20260929 01:55:23.435580 27257 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:23.436120 27256 heartbeater.cc:344] Connected to a master server at 127.0.71.126:41391
I20260929 01:55:23.436316 27256 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:23.436532 27257 heartbeater.cc:507] Master 127.0.71.125:35913 requested a full tablet report, sending...
I20260929 01:55:23.436875 27255 heartbeater.cc:344] Connected to a master server at 127.0.71.124:32895
I20260929 01:55:23.437106 27255 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:23.438262 27256 heartbeater.cc:507] Master 127.0.71.126:41391 requested a full tablet report, sending...
I20260929 01:55:23.438300 27255 heartbeater.cc:507] Master 127.0.71.124:32895 requested a full tablet report, sending...
I20260929 01:55:23.439399 27062 ts_manager.cc:194] Registered new tserver with Master: bc92f4e26ca24156a850ac6f0c03209c (127.0.71.65:36101)
I20260929 01:55:23.439420 26441 ts_manager.cc:194] Re-registered known tserver with Master: bc92f4e26ca24156a850ac6f0c03209c (127.0.71.65:36101)
I20260929 01:55:23.439656 26531 ts_manager.cc:194] Re-registered known tserver with Master: bc92f4e26ca24156a850ac6f0c03209c (127.0.71.65:36101)
I20260929 01:55:23.441421 26441 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:57767
I20260929 01:55:23.548868 27266 raft_consensus.cc:493] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader d6cf88e894de42e49d440d3b67403d37)
I20260929 01:55:23.549244 27266 raft_consensus.cc:515] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } }
I20260929 01:55:23.559760 26451 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "978f92c285a44324bb56bc9bf17fbce0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "38f49999d352471d8b1571385b4363c5" is_pre_election: true
I20260929 01:55:23.560177 27266 leader_election.cc:290] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers d6cf88e894de42e49d440d3b67403d37 (127.0.71.126:41391), 38f49999d352471d8b1571385b4363c5 (127.0.71.125:35913)
I20260929 01:55:23.566126 26877 heartbeater.cc:344] Connected to a master server at 127.0.71.126:41391
I20260929 01:55:23.566347 26877 heartbeater.cc:499] Master 127.0.71.126:41391 was elected leader, sending a full tablet report...
I20260929 01:55:23.572636 27062 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "d36e7a7040c34330b7346caf16813ae2" instance_seqno: 1790646921459431) as {username='kudu', principal='kudu/127.0.71.66@KRBTEST.COM'} at 127.0.71.66:41859; Asking this server to re-register.
I20260929 01:55:23.573041 27072 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "978f92c285a44324bb56bc9bf17fbce0" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d6cf88e894de42e49d440d3b67403d37" is_pre_election: true
I20260929 01:55:23.573385 27072 raft_consensus.cc:2393] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 978f92c285a44324bb56bc9bf17fbce0 in current term 3: Already voted for candidate 38f49999d352471d8b1571385b4363c5 in this term.
I20260929 01:55:23.574448 26877 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:23.574402 26517 leader_election.cc:304] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 978f92c285a44324bb56bc9bf17fbce0; no voters: 38f49999d352471d8b1571385b4363c5, d6cf88e894de42e49d440d3b67403d37
I20260929 01:55:23.574891 27266 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:55:23.575644 27062 ts_manager.cc:194] Registered new tserver with Master: d36e7a7040c34330b7346caf16813ae2 (127.0.71.66:40035)
I20260929 01:55:23.577080 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 27120
I20260929 01:55:23.577973 27266 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
W20260929 01:55:23.580292 27272 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:55:23.568046 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:55242 (local address 127.0.71.65:36101)
0929 01:55:23.572831 (+ 4785us) server_negotiation.cc:207] Beginning negotiation
0929 01:55:23.572841 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0929 01:55:23.573564 (+ 723us) server_negotiation.cc:408] Connection header received
0929 01:55:23.573644 (+ 80us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:55:23.573653 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:55:23.573942 (+ 289us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:55:23.574769 (+ 827us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:55:23.578663 (+ 3894us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:55242: BlockingRecv error: recv got EOF from 127.0.0.1:55242 (error 108)
Metrics: {"server-negotiator.queue_time_us":4622,"thread_start_us":6346,"threads_started":1}
W20260929 01:55:23.582975 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 27120: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:36101: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:55:23.583127 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 27120
I20260929 01:55:23.601694 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:36101
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--webserver_port=35583
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:41709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:55:23.731560 27072 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [term 3 FOLLOWER]: Refusing update from remote peer 38f49999d352471d8b1571385b4363c5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 01:55:23.732600 27128 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:55:23.738436 27275 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38f49999d352471d8b1571385b4363c5. Latest consensus state: current_term: 3 leader_uuid: "38f49999d352471d8b1571385b4363c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:23.738899 27275 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:55:23.748109 27275 sys_catalog.cc:455] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "38f49999d352471d8b1571385b4363c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:23.748387 27275 sys_catalog.cc:458] T 00000000000000000000000000000000 P d6cf88e894de42e49d440d3b67403d37 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:55:23.749984 27279 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "38f49999d352471d8b1571385b4363c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:23.750320 27279 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [sys.catalog]: This master's current role is: LEADER
I20260929 01:55:23.750600 26541 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [term 3 FOLLOWER]: Refusing update from remote peer 38f49999d352471d8b1571385b4363c5: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 01:55:23.751345 27279 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [LEADER]: Connected to new peer: Peer: permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:55:23.754805 27282 sys_catalog.cc:455] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "38f49999d352471d8b1571385b4363c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:23.755505 27282 sys_catalog.cc:458] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:55:23.755952 27266 sys_catalog.cc:455] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 38f49999d352471d8b1571385b4363c5. Latest consensus state: current_term: 3 leader_uuid: "38f49999d352471d8b1571385b4363c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:23.756147 27266 sys_catalog.cc:458] T 00000000000000000000000000000000 P 978f92c285a44324bb56bc9bf17fbce0 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:55:23.959992 27274 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:55:23.960484 27274 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:55:23.960677 27274 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:55:23.960783 27274 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:55:23.969628 27274 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:55:23.969916 27274 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:55:23.981350 27274 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:41709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:36101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=35583
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:55:23.982955 27274 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:55:23.984298 27274 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:55:23.994422 27288 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:23.994366 27287 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:23.994925 27290 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:23.995786 27274 server_base.cc:1061] running on GCE node
I20260929 01:55:23.996517 27274 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:55:23.997534 27274 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:55:23.998811 27274 hybrid_clock.cc:648] HybridClock initialized: now 1790646923998718 us; error 54 us; skew 500 ppm
I20260929 01:55:23.999222 27274 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:55:23 dist-test-slave-vz23 krb5kdc[26127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646923, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:55:24.003196 27274 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:55:24.005072 27274 webserver.cc:533] Webserver started at http://127.0.71.65:35583/ using document root <none> and password file <none>
I20260929 01:55:24.006085 27274 fs_manager.cc:362] Metadata directory not provided
I20260929 01:55:24.006196 27274 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:55:24.012399 27274 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 01:55:24.015750 27297 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:24.017481 27274 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:55:24.017696 27274 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "bc92f4e26ca24156a850ac6f0c03209c"
format_stamp: "Formatted at 2026-09-29 01:55:01 on dist-test-slave-vz23"
I20260929 01:55:24.018486 27274 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:55:24.043650 27274 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:55:24.047930 27274 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:55:24.048285 27274 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:55:24.049341 27274 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:55:24.051033 27274 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:55:24.051144 27274 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:24.051307 27274 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:55:24.051381 27274 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:55:24.051759 27274 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:55:24.051849 27274 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:55:24.051954 27274 fault_injection.cc:59] injecting fault for kudu (pid 27274): FLAGS_fault_before_start (process will exit)
I20260929 01:55:24.067780 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:36101
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--webserver_port=35583
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:41709
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:55:24.217245 27122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "38f49999d352471d8b1571385b4363c5" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d6cf88e894de42e49d440d3b67403d37" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41391 } } peers { permanent_uuid: "38f49999d352471d8b1571385b4363c5" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35913 } } peers { permanent_uuid: "978f92c285a44324bb56bc9bf17fbce0" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 32895 } } }
I20260929 01:55:24.217540 27122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 38f49999d352471d8b1571385b4363c5 [sys.catalog]: This master's current role is: LEADER
W20260929 01:55:24.438963 27305 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:55:24.439302 27305 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:55:24.439459 27305 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:55:24.448282 27305 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:55:24.448575 27305 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:55:24.459839 27305 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:41709
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:36101
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=35583
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:41391,127.0.71.125:35913,127.0.71.124:32895
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:55:24.461885 27305 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:55:24.463377 27305 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:55:24.475867 27310 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:24.475901 27311 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:24.477108 27305 server_base.cc:1061] running on GCE node
W20260929 01:55:24.475907 27313 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:24.478073 27305 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:55:24.479414 27305 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:55:24.480654 27305 hybrid_clock.cc:648] HybridClock initialized: now 1790646924480572 us; error 69 us; skew 500 ppm
I20260929 01:55:24.481024 27305 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:55:24 dist-test-slave-vz23 krb5kdc[26127](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646924, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:55:24.484862 27305 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:55:24.486683 27305 webserver.cc:533] Webserver started at http://127.0.71.65:35583/ using document root <none> and password file <none>
I20260929 01:55:24.487442 27305 fs_manager.cc:362] Metadata directory not provided
I20260929 01:55:24.487536 27305 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:55:24.493731 27305 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.002s
I20260929 01:55:24.497056 27320 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:24.498558 27305 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.002s
I20260929 01:55:24.498761 27305 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "bc92f4e26ca24156a850ac6f0c03209c"
format_stamp: "Formatted at 2026-09-29 01:55:01 on dist-test-slave-vz23"
I20260929 01:55:24.499574 27305 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:55:24.522403 27305 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:55:24.526611 27305 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:55:24.526908 27305 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:55:24.527909 27305 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:55:24.529657 27305 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:55:24.529778 27305 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:24.529904 27305 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:55:24.529999 27305 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:24.537623 26878 heartbeater.cc:499] Master 127.0.71.125:35913 was elected leader, sending a full tablet report...
I20260929 01:55:24.566540 27305 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:36101
I20260929 01:55:24.566627 27432 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:36101 every 8 connection(s)
I20260929 01:55:24.568133 27305 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:55:24.574836 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 27305
I20260929 01:55:24.575193 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 27305
W20260929 01:55:24.601874 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 27305: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:36101: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:55:24.602056 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 27305
I20260929 01:55:24.618569 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 26748
W20260929 01:55:24.619333 27437 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:55:24.571674 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:51191 (local address 127.0.71.124:32895)
0929 01:55:24.572174 (+ 500us) server_negotiation.cc:207] Beginning negotiation
0929 01:55:24.572182 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0929 01:55:24.599433 (+ 27251us) server_negotiation.cc:408] Connection header received
0929 01:55:24.599605 (+ 172us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:55:24.599612 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:55:24.599734 (+ 122us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:55:24.599872 (+ 138us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:55:24.618342 (+ 18470us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:51191: BlockingRecv error: recv got EOF from 127.0.71.65:51191 (error 108)
Metrics: {"server-negotiator.queue_time_us":372,"thread_start_us":212,"threads_started":1}
W20260929 01:55:24.619346 27442 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:55:24.572750 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:37371 (local address 127.0.71.125:35913)
0929 01:55:24.600838 (+ 28088us) server_negotiation.cc:207] Beginning negotiation
0929 01:55:24.600846 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0929 01:55:24.600868 (+ 22us) server_negotiation.cc:408] Connection header received
0929 01:55:24.600927 (+ 59us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:55:24.600933 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:55:24.601004 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:55:24.601112 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:55:24.618325 (+ 17213us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:37371: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":27970,"thread_start_us":27866,"threads_started":1}
W20260929 01:55:24.619997 27438 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:55:24.572485 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:57947 (local address 127.0.71.126:41391)
0929 01:55:24.601583 (+ 29098us) server_negotiation.cc:207] Beginning negotiation
0929 01:55:24.601589 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0929 01:55:24.601609 (+ 20us) server_negotiation.cc:408] Connection header received
0929 01:55:24.601667 (+ 58us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:55:24.601673 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:55:24.601748 (+ 75us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:55:24.601871 (+ 123us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:55:24.618373 (+ 16502us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:57947: BlockingRecv error: recv got EOF from 127.0.71.65:57947 (error 108)
Metrics: {"server-negotiator.queue_time_us":28990,"thread_start_us":28889,"threads_started":1}
W20260929 01:55:24.621694 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 26748: Not authorized: Client connection negotiation failed: client connection to 127.0.71.66:40035: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:55:24.621853 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 26748
I20260929 01:55:24.638049 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 26890
W20260929 01:55:24.640833 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 26890: Not authorized: Client connection negotiation failed: client connection to 127.0.71.67:34769: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:55:24.641017 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 26890
I20260929 01:55:24.657676 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 27029
W20260929 01:55:24.659950 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 27029: Not authorized: Client connection negotiation failed: client connection to 127.0.71.126:41391: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:55:24.660095 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 27029
W20260929 01:55:24.660106 27438 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:55:24.658722 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:40910 (local address 127.0.71.126:41391)
0929 01:55:24.658895 (+ 173us) server_negotiation.cc:207] Beginning negotiation
0929 01:55:24.658900 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0929 01:55:24.658916 (+ 16us) server_negotiation.cc:408] Connection header received
0929 01:55:24.658976 (+ 60us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:55:24.658980 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:55:24.659044 (+ 64us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:55:24.659133 (+ 89us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:55:24.659852 (+ 719us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40910: BlockingRecv error: recv got EOF from 127.0.0.1:40910 (error 108)
Metrics: {"server-negotiator.queue_time_us":62}
I20260929 01:55:24.681442 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 26409
I20260929 01:55:24.682814 26471 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:55:24.798418 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 26409
I20260929 01:55:24.819676 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 26499
I20260929 01:55:24.821062 26561 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:55:24.932955 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 26499
Shutting down hive metastore.
2026-09-29T01:55:25Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestBasicOperation/10 (47457 ms)
[ RUN ] ExternalMiniClusterTest.TestBasicOperation/11
Loading random data
Initializing database '/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:55:25 dist-test-slave-vz23 krb5kdc[27455](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:55:25 dist-test-slave-vz23 krb5kdc[27455](info): set up 2 sockets
Sep 29 01:55:25 dist-test-slave-vz23 krb5kdc[27455](info): commencing operation
krb5kdc: starting...
W20260929 01:55:27.465368 285 mini_kdc.cc:121] Time spent starting KDC: real 2.100s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 01:55:27 dist-test-slave-vz23 krb5kdc[27455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:55:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:55:27Z Disabled control of system clock
2026-09-29T01:55:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:55:27Z Disabled control of system clock
2026-09-29T01:55:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:55:27Z Disabled control of system clock
2026-09-29T01:55:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:55:27Z Disabled control of system clock
2026-09-29T01:55:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:55:27Z Disabled control of system clock
WARNING: no policy specified for hive/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "hive/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/hive_127.0.0.1.keytab.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/hive_127.0.0.1.keytab.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5.conf
2026-09-29 01:55:35: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
Sep 29 01:55:38 dist-test-slave-vz23 krb5kdc[27455](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790646938, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 01:55:44 dist-test-slave-vz23 krb5kdc[27455](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790646944, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 01:55:44 dist-test-slave-vz23 krb5kdc[27455](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790646944, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:55:46.404199 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:44289
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:40297,127.0.71.85:38077,127.0.71.86:36597,127.0.71.87:37187,127.0.71.88:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:55:46.768045 27759 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:55:46.768431 27759 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:55:46.768643 27759 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:55:46.777637 27759 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:55:46.777751 27759 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:55:46.777858 27759 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:55:46.777966 27759 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:55:46.789572 27759 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:40297,127.0.71.85:38077,127.0.71.86:36597,127.0.71.87:37187,127.0.71.88:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:44289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:55:46.791208 27759 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:55:46.792606 27759 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:55:46.801148 27764 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:46.801148 27765 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:46.801790 27767 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:46.802244 27759 server_base.cc:1061] running on GCE node
I20260929 01:55:46.802978 27759 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:55:46.804557 27759 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:55:46.805804 27759 hybrid_clock.cc:648] HybridClock initialized: now 1790646946805691 us; error 80 us; skew 500 ppm
I20260929 01:55:46.806176 27759 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:55:46 dist-test-slave-vz23 krb5kdc[27455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:55:46.809355 27759 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:55:46.810892 27759 webserver.cc:533] Webserver started at http://127.0.71.126:42871/ using document root <none> and password file <none>
I20260929 01:55:46.811583 27759 fs_manager.cc:362] Metadata directory not provided
I20260929 01:55:46.811679 27759 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:55:46.811946 27759 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:55:46.814345 27759 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "e128f9c5227a486482933c152890ec12"
format_stamp: "Formatted at 2026-09-29 01:55:46 on dist-test-slave-vz23"
I20260929 01:55:46.815059 27759 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "e128f9c5227a486482933c152890ec12"
format_stamp: "Formatted at 2026-09-29 01:55:46 on dist-test-slave-vz23"
I20260929 01:55:46.820549 27759 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 01:55:46.823887 27774 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:46.825301 27759 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:55:46.825439 27759 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "e128f9c5227a486482933c152890ec12"
format_stamp: "Formatted at 2026-09-29 01:55:46 on dist-test-slave-vz23"
I20260929 01:55:46.825598 27759 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:55:46.851779 27759 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:55:46.856055 27759 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:55:46.856299 27759 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:55:46.878129 27759 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:44289
I20260929 01:55:46.878151 27825 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:44289 every 8 connection(s)
I20260929 01:55:46.879674 27759 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:55:46.884138 27826 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:55:46.889580 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 27759
I20260929 01:55:46.889784 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:55:46.890309 27826 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:55:46 dist-test-slave-vz23 krb5kdc[27455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.125@KRBTEST.COM; defaulting to no policy
I20260929 01:55:46.918005 27826 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.125@KRBTEST.COM" created.
W20260929 01:55:46.919572 27777 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.125:38709: connect: Connection refused (error 111)
W20260929 01:55:46.921721 27826 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:38709: Network error: Client connection negotiation failed: client connection to 127.0.71.125:38709: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.125@KRBTEST.COM" created.
I20260929 01:55:46.971095 27826 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } attempt: 1
W20260929 01:55:46.973333 27826 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:38709: Network error: Client connection negotiation failed: client connection to 127.0.71.125:38709: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:55:47.003690 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:38709
--webserver_interface=127.0.71.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:40297,127.0.71.85:38077,127.0.71.86:36597,127.0.71.87:37187,127.0.71.88:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:55:47.041599 27826 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } attempt: 2
W20260929 01:55:47.044116 27826 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:38709: Network error: Client connection negotiation failed: client connection to 127.0.71.125:38709: connect: Connection refused (error 111)
I20260929 01:55:47.135425 27826 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } attempt: 3
W20260929 01:55:47.137651 27826 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:38709: Network error: Client connection negotiation failed: client connection to 127.0.71.125:38709: connect: Connection refused (error 111)
I20260929 01:55:47.280941 27826 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } attempt: 4
W20260929 01:55:47.283360 27826 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:38709: Network error: Client connection negotiation failed: client connection to 127.0.71.125:38709: connect: Connection refused (error 111)
W20260929 01:55:47.364655 27835 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:55:47.364986 27835 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:55:47.365070 27835 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:55:47.373721 27835 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:55:47.373833 27835 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:55:47.373891 27835 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:55:47.373939 27835 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:55:47.385017 27835 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:40297,127.0.71.85:38077,127.0.71.86:36597,127.0.71.87:37187,127.0.71.88:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:38709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:55:47.386588 27835 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:55:47.387894 27835 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:55:47.396939 27840 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:47.396999 27841 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:47.397806 27843 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:47.398545 27835 server_base.cc:1061] running on GCE node
I20260929 01:55:47.399432 27835 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:55:47.401386 27835 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:55:47.402750 27835 hybrid_clock.cc:648] HybridClock initialized: now 1790646947402611 us; error 101 us; skew 500 ppm
I20260929 01:55:47.403151 27835 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:55:47 dist-test-slave-vz23 krb5kdc[27455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:55:47.406525 27835 init.cc:377] Logged in from keytab as kudu/127.0.71.125@KRBTEST.COM (short username kudu)
I20260929 01:55:47.408735 27835 webserver.cc:533] Webserver started at http://127.0.71.125:41891/ using document root <none> and password file <none>
I20260929 01:55:47.409433 27835 fs_manager.cc:362] Metadata directory not provided
I20260929 01:55:47.409562 27835 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:55:47.409940 27835 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:55:47.412385 27835 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/data/instance:
uuid: "d0620fc4eb154a439efacce291731d63"
format_stamp: "Formatted at 2026-09-29 01:55:47 on dist-test-slave-vz23"
I20260929 01:55:47.413273 27835 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/wal/instance:
uuid: "d0620fc4eb154a439efacce291731d63"
format_stamp: "Formatted at 2026-09-29 01:55:47 on dist-test-slave-vz23"
I20260929 01:55:47.418883 27835 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 01:55:47.422500 27850 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:47.423993 27835 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:55:47.424196 27835 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "d0620fc4eb154a439efacce291731d63"
format_stamp: "Formatted at 2026-09-29 01:55:47 on dist-test-slave-vz23"
I20260929 01:55:47.424393 27835 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:55:47.448333 27835 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:55:47.452760 27835 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:55:47.453058 27835 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:55:47.474979 27835 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:38709
I20260929 01:55:47.474990 27901 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:38709 every 8 connection(s)
I20260929 01:55:47.476533 27835 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:55:47.480933 27902 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:55:47.481166 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 27835
I20260929 01:55:47.481375 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:55:47.490005 27902 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:55:47 dist-test-slave-vz23 krb5kdc[27455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.124@KRBTEST.COM" created.
I20260929 01:55:47.515239 27902 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:55:47 dist-test-slave-vz23 krb5kdc[27455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:55:47.534212 27902 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:55:47.535415 27853 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36311: connect: Connection refused (error 111)
W20260929 01:55:47.537276 27902 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36311: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36311: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.124@KRBTEST.COM" created.
I20260929 01:55:47.582659 27826 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } attempt: 5
I20260929 01:55:47.586643 27902 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } attempt: 1
W20260929 01:55:47.588965 27902 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36311: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36311: connect: Connection refused (error 111)
Sep 29 01:55:47 dist-test-slave-vz23 krb5kdc[27455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:55:47.598019 27826 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:55:47.600206 27826 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36311: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36311: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:55:47.608884 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:36311
--webserver_interface=127.0.71.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:40297,127.0.71.85:38077,127.0.71.86:36597,127.0.71.87:37187,127.0.71.88:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:55:47.651497 27826 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } attempt: 1
W20260929 01:55:47.653743 27826 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36311: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36311: connect: Connection refused (error 111)
I20260929 01:55:47.657197 27902 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } attempt: 2
W20260929 01:55:47.659169 27902 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36311: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36311: connect: Connection refused (error 111)
I20260929 01:55:47.722020 27826 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } attempt: 2
W20260929 01:55:47.724321 27826 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36311: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36311: connect: Connection refused (error 111)
I20260929 01:55:47.750427 27902 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } attempt: 3
W20260929 01:55:47.752761 27902 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36311: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36311: connect: Connection refused (error 111)
I20260929 01:55:47.830631 27826 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } attempt: 3
W20260929 01:55:47.833235 27826 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36311: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36311: connect: Connection refused (error 111)
I20260929 01:55:47.896044 27902 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } attempt: 4
W20260929 01:55:47.898334 27902 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36311: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36311: connect: Connection refused (error 111)
W20260929 01:55:47.954326 27912 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:55:47.954751 27912 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:55:47.954916 27912 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:55:47.963333 27912 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:55:47.963491 27912 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:55:47.963629 27912 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:55:47.963721 27912 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:55:47.975175 27912 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:40297,127.0.71.85:38077,127.0.71.86:36597,127.0.71.87:37187,127.0.71.88:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:36311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:55:47.976869 27912 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:55:47.978259 27912 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:55:47.987118 27917 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:47.987118 27918 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:47.987766 27920 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:47.988432 27912 server_base.cc:1061] running on GCE node
I20260929 01:55:47.989377 27912 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:55:47.991132 27912 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:55:47.992383 27912 hybrid_clock.cc:648] HybridClock initialized: now 1790646947992221 us; error 128 us; skew 500 ppm
I20260929 01:55:47.992823 27912 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:55:47 dist-test-slave-vz23 krb5kdc[27455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:55:47.996383 27912 init.cc:377] Logged in from keytab as kudu/127.0.71.124@KRBTEST.COM (short username kudu)
I20260929 01:55:47.998180 27912 webserver.cc:533] Webserver started at http://127.0.71.124:44657/ using document root <none> and password file <none>
I20260929 01:55:47.998904 27912 fs_manager.cc:362] Metadata directory not provided
I20260929 01:55:47.999011 27912 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:55:47.999320 27912 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:55:48.001870 27912 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/data/instance:
uuid: "651d58de18b84e00a50539ad4f986859"
format_stamp: "Formatted at 2026-09-29 01:55:47 on dist-test-slave-vz23"
I20260929 01:55:48.002738 27912 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/wal/instance:
uuid: "651d58de18b84e00a50539ad4f986859"
format_stamp: "Formatted at 2026-09-29 01:55:47 on dist-test-slave-vz23"
I20260929 01:55:48.008432 27912 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 01:55:48.010460 27826 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } attempt: 4
I20260929 01:55:48.012265 27927 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
W20260929 01:55:48.012672 27826 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36311: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36311: connect: Connection refused (error 111)
I20260929 01:55:48.013935 27912 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:55:48.014125 27912 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "651d58de18b84e00a50539ad4f986859"
format_stamp: "Formatted at 2026-09-29 01:55:47 on dist-test-slave-vz23"
I20260929 01:55:48.014344 27912 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:55:48.033430 27912 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:55:48.038579 27912 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:55:48.038899 27912 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:55:48.060282 27912 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:36311
I20260929 01:55:48.060305 27978 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:36311 every 8 connection(s)
I20260929 01:55:48.062040 27912 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:55:48.062686 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 27912
I20260929 01:55:48.062889 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:55:48.067368 27979 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 01:55:48 dist-test-slave-vz23 krb5kdc[27455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:55:48.075623 27979 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:55:48 dist-test-slave-vz23 krb5kdc[27455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:55:48 dist-test-slave-vz23 krb5kdc[27455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:55:48.099152 27979 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:55:48 dist-test-slave-vz23 krb5kdc[27455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:55:48 dist-test-slave-vz23 krb5kdc[27455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:55:48.114275 27979 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:55:48 dist-test-slave-vz23 krb5kdc[27455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:55:48.134142 27979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859: Bootstrap starting.
I20260929 01:55:48.137545 27979 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859: Neither blocks nor log segments found. Creating new log.
I20260929 01:55:48.138633 27979 log.cc:826] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859: Log is configured to *not* fsync() on all Append() calls
I20260929 01:55:48.141595 27979 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859: No bootstrap required, opened a new log
I20260929 01:55:48.146988 27979 raft_consensus.cc:359] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } }
I20260929 01:55:48.147394 27979 raft_consensus.cc:385] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:55:48.147574 27979 raft_consensus.cc:740] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 651d58de18b84e00a50539ad4f986859, State: Initialized, Role: FOLLOWER
I20260929 01:55:48.148281 27979 consensus_queue.cc:260] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } }
I20260929 01:55:48.149387 27987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:55:48.149731 27987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:55:48.150389 27979 sys_catalog.cc:565] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:55:48.160430 27998 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:55:48.160605 27998 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:55:48.157692 27912 master_runner.cc:187] Error getting master registration for 127.0.71.126:44289: OK, instance_id { permanent_uuid: "e128f9c5227a486482933c152890ec12" instance_seqno: 1790646946872375 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:55:48.163105 27912 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:55:48.197556 27902 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } attempt: 5
Sep 29 01:55:48 dist-test-slave-vz23 krb5kdc[27455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646947, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:55:48.216712 27902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63: Bootstrap starting.
I20260929 01:55:48.220336 27902 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63: Neither blocks nor log segments found. Creating new log.
I20260929 01:55:48.221637 27902 log.cc:826] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63: Log is configured to *not* fsync() on all Append() calls
I20260929 01:55:48.224094 27902 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63: No bootstrap required, opened a new log
I20260929 01:55:48.229691 27902 raft_consensus.cc:359] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } }
I20260929 01:55:48.229974 27902 raft_consensus.cc:385] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:55:48.230108 27902 raft_consensus.cc:740] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0620fc4eb154a439efacce291731d63, State: Initialized, Role: FOLLOWER
I20260929 01:55:48.230777 27902 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } }
I20260929 01:55:48.231807 28001 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:55:48.232101 28001 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:55:48.233021 27902 sys_catalog.cc:565] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:55:48.245675 28012 catalog_manager.cc:1607] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:55:48.245910 28012 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:55:48.242247 27835 master_runner.cc:187] Error getting master registration for 127.0.71.126:44289: OK, instance_id { permanent_uuid: "e128f9c5227a486482933c152890ec12" instance_seqno: 1790646946872375 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:55:48.249228 27835 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:55:48.289948 27826 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } attempt: 5
W20260929 01:55:48.291287 27933 tablet.cc:2433] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859: Can't schedule compaction. Clean time has not been advanced past its initial value.
Sep 29 01:55:48 dist-test-slave-vz23 krb5kdc[27455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646946, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:55:48.312814 27826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: Bootstrap starting.
I20260929 01:55:48.317009 27826 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: Neither blocks nor log segments found. Creating new log.
I20260929 01:55:48.318337 27826 log.cc:826] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: Log is configured to *not* fsync() on all Append() calls
I20260929 01:55:48.321524 27826 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: No bootstrap required, opened a new log
I20260929 01:55:48.329926 27826 raft_consensus.cc:359] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } }
I20260929 01:55:48.330341 27826 raft_consensus.cc:385] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:55:48.330482 27826 raft_consensus.cc:740] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e128f9c5227a486482933c152890ec12, State: Initialized, Role: FOLLOWER
I20260929 01:55:48.331209 27826 consensus_queue.cc:260] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } }
I20260929 01:55:48.332227 28015 sys_catalog.cc:455] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:55:48.332635 28015 sys_catalog.cc:458] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:55:48.333576 27826 sys_catalog.cc:565] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:55:48.359719 27780 tablet.cc:2433] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:55:48.362498 27759 master_runner.cc:264] No leader master found from master e128f9c5227a486482933c152890ec12
I20260929 01:55:48.362660 27759 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:55:48.363312 28026 catalog_manager.cc:1607] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:55:48.363457 28026 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 01:55:48.455873 27856 tablet.cc:2433] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:55:48.488996 27987 raft_consensus.cc:493] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:55:48.489300 27987 raft_consensus.cc:515] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } }
I20260929 01:55:48.490981 27987 leader_election.cc:290] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers e128f9c5227a486482933c152890ec12 (127.0.71.126:44289), d0620fc4eb154a439efacce291731d63 (127.0.71.125:38709)
I20260929 01:55:48.492772 27877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "651d58de18b84e00a50539ad4f986859" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0620fc4eb154a439efacce291731d63" is_pre_election: true
I20260929 01:55:48.492780 27801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "651d58de18b84e00a50539ad4f986859" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e128f9c5227a486482933c152890ec12" is_pre_election: true
I20260929 01:55:48.493265 27801 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 651d58de18b84e00a50539ad4f986859 in term 0.
I20260929 01:55:48.493263 27877 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 651d58de18b84e00a50539ad4f986859 in term 0.
I20260929 01:55:48.494122 27930 leader_election.cc:304] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 651d58de18b84e00a50539ad4f986859, d0620fc4eb154a439efacce291731d63; no voters:
I20260929 01:55:48.495170 27987 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:55:48.495324 27987 raft_consensus.cc:493] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:55:48.495419 27987 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:55:48.498899 27987 raft_consensus.cc:515] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } }
I20260929 01:55:48.500149 27801 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "651d58de18b84e00a50539ad4f986859" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e128f9c5227a486482933c152890ec12"
I20260929 01:55:48.500365 27801 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:55:48.504138 27987 leader_election.cc:290] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [CANDIDATE]: Term 1 election: Requested vote from peers e128f9c5227a486482933c152890ec12 (127.0.71.126:44289), d0620fc4eb154a439efacce291731d63 (127.0.71.125:38709)
I20260929 01:55:48.504823 27877 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "651d58de18b84e00a50539ad4f986859" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d0620fc4eb154a439efacce291731d63"
I20260929 01:55:48.505033 27877 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:55:48.507499 27801 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 651d58de18b84e00a50539ad4f986859 in term 1.
I20260929 01:55:48.509068 27929 leader_election.cc:304] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 651d58de18b84e00a50539ad4f986859, e128f9c5227a486482933c152890ec12; no voters:
I20260929 01:55:48.509495 27877 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 651d58de18b84e00a50539ad4f986859 in term 1.
I20260929 01:55:48.510288 27987 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:55:48.518654 27987 raft_consensus.cc:697] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [term 1 LEADER]: Becoming Leader. State: Replica: 651d58de18b84e00a50539ad4f986859, State: Running, Role: LEADER
I20260929 01:55:48.519451 27987 consensus_queue.cc:237] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } }
I20260929 01:55:48.527663 28028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 651d58de18b84e00a50539ad4f986859. Latest consensus state: current_term: 1 leader_uuid: "651d58de18b84e00a50539ad4f986859" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:55:48.528134 28028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [sys.catalog]: This master's current role is: LEADER
I20260929 01:55:48.538802 28030 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:55:48.541318 28030 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:55:48.557143 27877 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [term 1 FOLLOWER]: Refusing update from remote peer 651d58de18b84e00a50539ad4f986859: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:55:48.558362 28028 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:55:48.569872 27801 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [term 1 FOLLOWER]: Refusing update from remote peer 651d58de18b84e00a50539ad4f986859: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:55:48.571069 28028 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:55:48.591475 28001 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 651d58de18b84e00a50539ad4f986859. Latest consensus state: current_term: 1 leader_uuid: "651d58de18b84e00a50539ad4f986859" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:55:48.591784 28001 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:55:48.595309 28031 mvcc.cc:204] Tried to move back new op lower bound from 7334489901276655616 to 7334489901141610496. Current Snapshot: MvccSnapshot[applied={T|T < 7334489901276655616}]
I20260929 01:55:48.604698 27987 sys_catalog.cc:455] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "651d58de18b84e00a50539ad4f986859" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:55:48.605010 27987 sys_catalog.cc:458] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [sys.catalog]: This master's current role is: LEADER
I20260929 01:55:48.607966 28030 catalog_manager.cc:1396] Generated new cluster ID: 63a1f8021ce442519704c5b15912a2a3
I20260929 01:55:48.608207 28030 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:55:48.610235 28001 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "651d58de18b84e00a50539ad4f986859" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:55:48.610457 28001 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:55:48.620198 28015 sys_catalog.cc:455] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 651d58de18b84e00a50539ad4f986859. Latest consensus state: current_term: 1 leader_uuid: "651d58de18b84e00a50539ad4f986859" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:55:48.620543 28015 sys_catalog.cc:458] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:55:48.625878 28028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "651d58de18b84e00a50539ad4f986859" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:55:48.626163 28028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [sys.catalog]: This master's current role is: LEADER
I20260929 01:55:48.633038 28015 sys_catalog.cc:455] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "651d58de18b84e00a50539ad4f986859" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:55:48.633271 28015 sys_catalog.cc:458] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:55:48.667268 28030 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 01:55:48.669292 28030 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:55:48.685184 28030 catalog_manager.cc:6219] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859: Generated new TSK 0
I20260929 01:55:48.686375 28030 catalog_manager.cc:1563] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:55:48.837246 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:40297,127.0.71.85:38077,127.0.71.86:36597,127.0.71.87:37187,127.0.71.88:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:55:49.215858 28044 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:55:49.216180 28044 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:55:49.216274 28044 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:55:49.224902 28044 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:55:49.225085 28044 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:55:49.236541 28044 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:40297,127.0.71.85:38077,127.0.71.86:36597,127.0.71.87:37187,127.0.71.88:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:55:49.238145 28044 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:55:49.239603 28044 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:55:49.248860 28049 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:49.249315 28012 catalog_manager.cc:1308] Loaded cluster ID: 63a1f8021ce442519704c5b15912a2a3
I20260929 01:55:49.249423 28012 catalog_manager.cc:1601] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63: loading cluster ID for follower catalog manager: success
W20260929 01:55:49.250523 28050 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:49.251595 28052 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:49.252208 28044 server_base.cc:1061] running on GCE node
I20260929 01:55:49.253087 28044 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:55:49.254482 28044 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:55:49.254468 28012 catalog_manager.cc:1623] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63: acquiring CA information for follower catalog manager: success
I20260929 01:55:49.256870 28012 catalog_manager.cc:1651] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:55:49.257833 28044 hybrid_clock.cc:648] HybridClock initialized: now 1790646949257668 us; error 130 us; skew 500 ppm
I20260929 01:55:49.258177 28044 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:55:49 dist-test-slave-vz23 krb5kdc[27455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:55:49.261686 28044 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:55:49.263375 28044 webserver.cc:533] Webserver started at http://127.0.71.65:36577/ using document root <none> and password file <none>
I20260929 01:55:49.264187 28044 fs_manager.cc:362] Metadata directory not provided
I20260929 01:55:49.264287 28044 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:55:49.264662 28044 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:55:49.266973 28044 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "a14c7d2a0afb4161926cd8cce4fbde4e"
format_stamp: "Formatted at 2026-09-29 01:55:49 on dist-test-slave-vz23"
I20260929 01:55:49.267670 28044 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "a14c7d2a0afb4161926cd8cce4fbde4e"
format_stamp: "Formatted at 2026-09-29 01:55:49 on dist-test-slave-vz23"
I20260929 01:55:49.273131 28044 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260929 01:55:49.276654 28060 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:49.278051 28044 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:55:49.278229 28044 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "a14c7d2a0afb4161926cd8cce4fbde4e"
format_stamp: "Formatted at 2026-09-29 01:55:49 on dist-test-slave-vz23"
I20260929 01:55:49.278424 28044 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:55:49.295300 28044 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:55:49.299494 28044 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:55:49.299814 28044 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:55:49.300837 28044 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:55:49.302547 28044 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:55:49.302661 28044 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:49.302820 28044 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:55:49.302894 28044 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:49.339949 28044 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:41063
I20260929 01:55:49.340001 28172 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:41063 every 8 connection(s)
I20260929 01:55:49.341600 28044 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:55:49.342393 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 28044
I20260929 01:55:49.342684 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
I20260929 01:55:49.366026 28026 catalog_manager.cc:1308] Loaded cluster ID: 63a1f8021ce442519704c5b15912a2a3
I20260929 01:55:49.366180 28026 catalog_manager.cc:1601] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: loading cluster ID for follower catalog manager: success
I20260929 01:55:49.370316 28026 catalog_manager.cc:1623] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: acquiring CA information for follower catalog manager: success
I20260929 01:55:49.373111 28026 catalog_manager.cc:1651] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Sep 29 01:55:49 dist-test-slave-vz23 krb5kdc[27455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:55:49 dist-test-slave-vz23 krb5kdc[27455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:55:49 dist-test-slave-vz23 krb5kdc[27455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.66@KRBTEST.COM" created.
I20260929 01:55:49.395917 28174 heartbeater.cc:344] Connected to a master server at 127.0.71.126:44289
I20260929 01:55:49.396359 28174 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:49.396775 28173 heartbeater.cc:344] Connected to a master server at 127.0.71.124:36311
I20260929 01:55:49.396945 28173 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:49.397171 28175 heartbeater.cc:344] Connected to a master server at 127.0.71.125:38709
I20260929 01:55:49.397349 28175 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:49.397631 28174 heartbeater.cc:507] Master 127.0.71.126:44289 requested a full tablet report, sending...
I20260929 01:55:49.397662 28173 heartbeater.cc:507] Master 127.0.71.124:36311 requested a full tablet report, sending...
I20260929 01:55:49.397903 28175 heartbeater.cc:507] Master 127.0.71.125:38709 requested a full tablet report, sending...
I20260929 01:55:49.400281 27791 ts_manager.cc:194] Registered new tserver with Master: a14c7d2a0afb4161926cd8cce4fbde4e (127.0.71.65:41063)
I20260929 01:55:49.400260 27944 ts_manager.cc:194] Registered new tserver with Master: a14c7d2a0afb4161926cd8cce4fbde4e (127.0.71.65:41063)
I20260929 01:55:49.401868 27867 ts_manager.cc:194] Registered new tserver with Master: a14c7d2a0afb4161926cd8cce4fbde4e (127.0.71.65:41063)
I20260929 01:55:49.402060 27944 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:58525
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.66@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.66@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.66 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.66 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:55:49.480700 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.66:0
--local_ip_for_outbound_sockets=127.0.71.66
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:40297,127.0.71.85:38077,127.0.71.86:36597,127.0.71.87:37187,127.0.71.88:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:55:49.828246 28187 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:55:49.828627 28187 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:55:49.828732 28187 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:55:49.837497 28187 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:55:49.837685 28187 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.66
I20260929 01:55:49.849086 28187 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:40297,127.0.71.85:38077,127.0.71.86:36597,127.0.71.87:37187,127.0.71.88:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.71.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:55:49.850674 28187 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:55:49.852010 28187 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:55:49.862169 28193 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:49.862911 28192 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:49.862958 28195 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:49.863445 28187 server_base.cc:1061] running on GCE node
I20260929 01:55:49.864012 28187 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:55:49.865389 28187 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:55:49.866755 28187 hybrid_clock.cc:648] HybridClock initialized: now 1790646949866599 us; error 124 us; skew 500 ppm
I20260929 01:55:49.867115 28187 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:55:49 dist-test-slave-vz23 krb5kdc[27455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:55:49.870568 28187 init.cc:377] Logged in from keytab as kudu/127.0.71.66@KRBTEST.COM (short username kudu)
I20260929 01:55:49.872272 28187 webserver.cc:533] Webserver started at http://127.0.71.66:45289/ using document root <none> and password file <none>
I20260929 01:55:49.873042 28187 fs_manager.cc:362] Metadata directory not provided
I20260929 01:55:49.873162 28187 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:55:49.873447 28187 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:55:49.875831 28187 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/data/instance:
uuid: "ee03cbc827f945e58402b9cfa92b92e4"
format_stamp: "Formatted at 2026-09-29 01:55:49 on dist-test-slave-vz23"
I20260929 01:55:49.876655 28187 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/wal/instance:
uuid: "ee03cbc827f945e58402b9cfa92b92e4"
format_stamp: "Formatted at 2026-09-29 01:55:49 on dist-test-slave-vz23"
I20260929 01:55:49.882328 28187 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 01:55:49.886379 28202 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:49.887861 28187 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:55:49.888043 28187 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/wal
uuid: "ee03cbc827f945e58402b9cfa92b92e4"
format_stamp: "Formatted at 2026-09-29 01:55:49 on dist-test-slave-vz23"
I20260929 01:55:49.888284 28187 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:55:49.917593 28187 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:55:49.922283 28187 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:55:49.922590 28187 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:55:49.923548 28187 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:55:49.925163 28187 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:55:49.925269 28187 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:49.925424 28187 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:55:49.925498 28187 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:49.961125 28187 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.66:42189
I20260929 01:55:49.961182 28314 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.66:42189 every 8 connection(s)
I20260929 01:55:49.962594 28187 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
I20260929 01:55:49.966503 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 28187
I20260929 01:55:49.966704 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/wal/instance
Sep 29 01:55:49 dist-test-slave-vz23 krb5kdc[27455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:55:49 dist-test-slave-vz23 krb5kdc[27455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:55:49 dist-test-slave-vz23 krb5kdc[27455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646949, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:55:49.996801 28315 heartbeater.cc:344] Connected to a master server at 127.0.71.124:36311
I20260929 01:55:49.997146 28315 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:49.998265 28315 heartbeater.cc:507] Master 127.0.71.124:36311 requested a full tablet report, sending...
I20260929 01:55:50.000378 27944 ts_manager.cc:194] Registered new tserver with Master: ee03cbc827f945e58402b9cfa92b92e4 (127.0.71.66:42189)
I20260929 01:55:50.001303 28316 heartbeater.cc:344] Connected to a master server at 127.0.71.126:44289
I20260929 01:55:50.001466 28316 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:50.001552 27944 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.66@KRBTEST.COM'} at 127.0.71.66:47397
I20260929 01:55:50.002079 28316 heartbeater.cc:507] Master 127.0.71.126:44289 requested a full tablet report, sending...
I20260929 01:55:50.003445 27791 ts_manager.cc:194] Registered new tserver with Master: ee03cbc827f945e58402b9cfa92b92e4 (127.0.71.66:42189)
WARNING: no policy specified for kudu/127.0.71.67@KRBTEST.COM; defaulting to no policy
I20260929 01:55:50.007728 28317 heartbeater.cc:344] Connected to a master server at 127.0.71.125:38709
I20260929 01:55:50.007860 28317 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:50.008172 28317 heartbeater.cc:507] Master 127.0.71.125:38709 requested a full tablet report, sending...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.67@KRBTEST.COM" created.
I20260929 01:55:50.009292 27867 ts_manager.cc:194] Registered new tserver with Master: ee03cbc827f945e58402b9cfa92b92e4 (127.0.71.66:42189)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.67@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.67@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.67 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.67 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:55:50.096516 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.67:0
--local_ip_for_outbound_sockets=127.0.71.67
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:40297,127.0.71.85:38077,127.0.71.86:36597,127.0.71.87:37187,127.0.71.88:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:55:50.405705 28173 heartbeater.cc:499] Master 127.0.71.124:36311 was elected leader, sending a full tablet report...
W20260929 01:55:50.444780 28329 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:55:50.445163 28329 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:55:50.445317 28329 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:55:50.454027 28329 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:55:50.454277 28329 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.67
I20260929 01:55:50.465050 28329 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:40297,127.0.71.85:38077,127.0.71.86:36597,127.0.71.87:37187,127.0.71.88:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.71.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:55:50.467052 28329 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:55:50.468384 28329 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:55:50.478348 28334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:50.478585 28335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:50.479561 28337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:50.480836 28329 server_base.cc:1061] running on GCE node
I20260929 01:55:50.481580 28329 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:55:50.483085 28329 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:55:50.484424 28329 hybrid_clock.cc:648] HybridClock initialized: now 1790646950484239 us; error 146 us; skew 500 ppm
I20260929 01:55:50.484833 28329 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:55:50 dist-test-slave-vz23 krb5kdc[27455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:55:50.489341 28329 init.cc:377] Logged in from keytab as kudu/127.0.71.67@KRBTEST.COM (short username kudu)
I20260929 01:55:50.491215 28329 webserver.cc:533] Webserver started at http://127.0.71.67:45721/ using document root <none> and password file <none>
I20260929 01:55:50.491926 28329 fs_manager.cc:362] Metadata directory not provided
I20260929 01:55:50.492069 28329 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:55:50.492413 28329 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:55:50.494860 28329 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/data/instance:
uuid: "bc0338288524439284a8f6b010ef98c1"
format_stamp: "Formatted at 2026-09-29 01:55:50 on dist-test-slave-vz23"
I20260929 01:55:50.495900 28329 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/wal/instance:
uuid: "bc0338288524439284a8f6b010ef98c1"
format_stamp: "Formatted at 2026-09-29 01:55:50 on dist-test-slave-vz23"
I20260929 01:55:50.502276 28329 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.001s sys 0.007s
I20260929 01:55:50.506685 28344 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:50.508538 28329 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 01:55:50.508741 28329 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/wal
uuid: "bc0338288524439284a8f6b010ef98c1"
format_stamp: "Formatted at 2026-09-29 01:55:50 on dist-test-slave-vz23"
I20260929 01:55:50.509002 28329 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:55:50.525123 28329 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:55:50.529454 28329 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:55:50.529855 28329 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:55:50.531270 28329 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:55:50.533102 28329 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:55:50.533223 28329 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:50.533367 28329 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:55:50.533478 28329 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:50.571346 28329 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.67:37973
I20260929 01:55:50.571386 28456 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.67:37973 every 8 connection(s)
I20260929 01:55:50.572793 28329 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
I20260929 01:55:50.582269 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 28329
I20260929 01:55:50.582497 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/wal/instance
Sep 29 01:55:50 dist-test-slave-vz23 krb5kdc[27455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:55:50 dist-test-slave-vz23 krb5kdc[27455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:55:50 dist-test-slave-vz23 krb5kdc[27455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646950, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:55:50.600533 28459 heartbeater.cc:344] Connected to a master server at 127.0.71.125:38709
I20260929 01:55:50.600972 28459 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:50.601755 28458 heartbeater.cc:344] Connected to a master server at 127.0.71.126:44289
I20260929 01:55:50.601857 28459 heartbeater.cc:507] Master 127.0.71.125:38709 requested a full tablet report, sending...
I20260929 01:55:50.601926 28458 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:50.602824 28457 heartbeater.cc:344] Connected to a master server at 127.0.71.124:36311
I20260929 01:55:50.602965 28457 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:50.602963 28458 heartbeater.cc:507] Master 127.0.71.126:44289 requested a full tablet report, sending...
I20260929 01:55:50.603511 28457 heartbeater.cc:507] Master 127.0.71.124:36311 requested a full tablet report, sending...
I20260929 01:55:50.604127 27791 ts_manager.cc:194] Registered new tserver with Master: bc0338288524439284a8f6b010ef98c1 (127.0.71.67:37973)
I20260929 01:55:50.604171 27867 ts_manager.cc:194] Registered new tserver with Master: bc0338288524439284a8f6b010ef98c1 (127.0.71.67:37973)
I20260929 01:55:50.604558 27944 ts_manager.cc:194] Registered new tserver with Master: bc0338288524439284a8f6b010ef98c1 (127.0.71.67:37973)
I20260929 01:55:50.605315 27944 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.67@KRBTEST.COM'} at 127.0.71.67:50215
I20260929 01:55:50.611186 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:55:50.618997 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:55:50.619138 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 27759
I20260929 01:55:50.620136 27821 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:55:50.731925 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 27759
I20260929 01:55:50.754387 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:44289
--webserver_interface=127.0.71.126
--webserver_port=42871
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:40297,127.0.71.85:38077,127.0.71.86:36597,127.0.71.87:37187,127.0.71.88:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:55:51.004330 28315 heartbeater.cc:499] Master 127.0.71.124:36311 was elected leader, sending a full tablet report...
W20260929 01:55:51.007895 28316 heartbeater.cc:646] Failed to heartbeat to 127.0.71.126:44289 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.71.126:44289: connect: Connection refused (error 111)
W20260929 01:55:51.100837 28468 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:55:51.101147 28468 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:55:51.101315 28468 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:55:51.108882 27929 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.126:44289: connect: Connection refused (error 111)
W20260929 01:55:51.110096 28468 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:55:51.110234 28468 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:55:51.110334 28468 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:55:51.110437 28468 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260929 01:55:51.110478 27929 consensus_peers.cc:597] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 -> Peer e128f9c5227a486482933c152890ec12 (127.0.71.126:44289): Couldn't send request to peer e128f9c5227a486482933c152890ec12. Status: Network error: Client connection negotiation failed: client connection to 127.0.71.126:44289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 01:55:51.121937 28468 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:40297,127.0.71.85:38077,127.0.71.86:36597,127.0.71.87:37187,127.0.71.88:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:44289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=42871
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:55:51.123610 28468 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:55:51.125041 28468 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:55:51.133646 28477 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:51.133670 28476 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:51.134799 28479 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:51.134960 28468 server_base.cc:1061] running on GCE node
I20260929 01:55:51.135846 28468 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:55:51.137531 28468 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:55:51.138789 28468 hybrid_clock.cc:648] HybridClock initialized: now 1790646951138625 us; error 129 us; skew 500 ppm
I20260929 01:55:51.139206 28468 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:55:51 dist-test-slave-vz23 krb5kdc[27455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:55:51.142762 28468 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:55:51.144587 28468 webserver.cc:533] Webserver started at http://127.0.71.126:42871/ using document root <none> and password file <none>
I20260929 01:55:51.145339 28468 fs_manager.cc:362] Metadata directory not provided
I20260929 01:55:51.145498 28468 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:55:51.151170 28468 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:55:51.154309 28486 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:51.155603 28468 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:55:51.155789 28468 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "e128f9c5227a486482933c152890ec12"
format_stamp: "Formatted at 2026-09-29 01:55:46 on dist-test-slave-vz23"
I20260929 01:55:51.156544 28468 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:55:51.183226 28468 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:55:51.187394 28468 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:55:51.187691 28468 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:55:51.208775 28468 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:44289
I20260929 01:55:51.208797 28537 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:44289 every 8 connection(s)
I20260929 01:55:51.210486 28468 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:55:51.211143 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 28468
I20260929 01:55:51.211530 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 28044
I20260929 01:55:51.218760 28538 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:55:51.222968 28538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: Bootstrap starting.
Sep 29 01:55:51 dist-test-slave-vz23 krb5kdc[27455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 01:55:51.234839 28168 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:55:51.242285 28538 log.cc:826] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: Log is configured to *not* fsync() on all Append() calls
I20260929 01:55:51.250808 28538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:55:51.251358 28538 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: Bootstrap complete.
I20260929 01:55:51.258333 28538 raft_consensus.cc:359] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } }
I20260929 01:55:51.259030 28538 raft_consensus.cc:740] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e128f9c5227a486482933c152890ec12, State: Initialized, Role: FOLLOWER
I20260929 01:55:51.259721 28538 consensus_queue.cc:260] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } }
I20260929 01:55:51.260583 28544 sys_catalog.cc:455] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:55:51.260902 28544 sys_catalog.cc:458] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:55:51.261651 28538 sys_catalog.cc:565] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:55:51.280390 28555 catalog_manager.cc:1308] Loaded cluster ID: 63a1f8021ce442519704c5b15912a2a3
I20260929 01:55:51.280535 28555 catalog_manager.cc:1601] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: loading cluster ID for follower catalog manager: success
I20260929 01:55:51.285318 28555 catalog_manager.cc:1623] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: acquiring CA information for follower catalog manager: success
Sep 29 01:55:51 dist-test-slave-vz23 krb5kdc[27455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:55:51.288422 28555 catalog_manager.cc:1651] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:55:51.348783 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 28044
I20260929 01:55:51.368426 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:41063
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--webserver_port=36577
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:40297,127.0.71.85:38077,127.0.71.86:36597,127.0.71.87:37187,127.0.71.88:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:55:51.607821 28457 heartbeater.cc:499] Master 127.0.71.124:36311 was elected leader, sending a full tablet report...
I20260929 01:55:51.617308 28503 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "bc0338288524439284a8f6b010ef98c1" instance_seqno: 1790646950563227) as {username='kudu', principal='kudu/127.0.71.67@KRBTEST.COM'} at 127.0.71.67:52239; Asking this server to re-register.
I20260929 01:55:51.618182 28458 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:51.620010 28503 ts_manager.cc:194] Registered new tserver with Master: bc0338288524439284a8f6b010ef98c1 (127.0.71.67:37973)
I20260929 01:55:51.635609 28054 sys_catalog.cc:455] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "651d58de18b84e00a50539ad4f986859" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:55:51.635928 28054 sys_catalog.cc:458] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [sys.catalog]: This master's current role is: LEADER
I20260929 01:55:51.649750 28544 sys_catalog.cc:455] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 651d58de18b84e00a50539ad4f986859. Latest consensus state: current_term: 1 leader_uuid: "651d58de18b84e00a50539ad4f986859" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:55:51.650085 28544 sys_catalog.cc:458] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:55:51.725461 28561 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:55:51.725811 28561 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:55:51.725973 28561 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:55:51.734632 28561 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:55:51.734865 28561 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:55:51.745873 28561 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:40297,127.0.71.85:38077,127.0.71.86:36597,127.0.71.87:37187,127.0.71.88:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:41063
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=36577
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:55:51.747519 28561 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:55:51.748950 28561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:55:51.759042 28572 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:51.759231 28574 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:51.759023 28571 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:51.760846 28561 server_base.cc:1061] running on GCE node
I20260929 01:55:51.761445 28561 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:55:51.762729 28561 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:55:51.764072 28561 hybrid_clock.cc:648] HybridClock initialized: now 1790646951763930 us; error 104 us; skew 500 ppm
I20260929 01:55:51.764530 28561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:55:51 dist-test-slave-vz23 krb5kdc[27455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:55:51.768422 28561 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:55:51.770263 28561 webserver.cc:533] Webserver started at http://127.0.71.65:36577/ using document root <none> and password file <none>
I20260929 01:55:51.771062 28561 fs_manager.cc:362] Metadata directory not provided
I20260929 01:55:51.771209 28561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:55:51.777352 28561 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:55:51.780333 28581 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:51.781978 28561 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:55:51.782192 28561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "a14c7d2a0afb4161926cd8cce4fbde4e"
format_stamp: "Formatted at 2026-09-29 01:55:49 on dist-test-slave-vz23"
I20260929 01:55:51.782990 28561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:55:51.806610 28561 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:55:51.815835 28561 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:55:51.816188 28561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:55:51.817361 28561 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:55:51.819136 28561 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:55:51.819269 28561 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:51.819419 28561 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:55:51.819519 28561 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:51.858652 28561 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:41063
I20260929 01:55:51.858750 28693 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:41063 every 8 connection(s)
I20260929 01:55:51.860427 28561 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:55:51.867612 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 28561
Sep 29 01:55:51 dist-test-slave-vz23 krb5kdc[27455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646927, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for hive/127.0.0.1@KRBTEST.COM
Sep 29 01:55:51 dist-test-slave-vz23 krb5kdc[27455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:55:51 dist-test-slave-vz23 krb5kdc[27455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:55:51 dist-test-slave-vz23 krb5kdc[27455](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646951, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:55:51.895462 28696 heartbeater.cc:344] Connected to a master server at 127.0.71.125:38709
I20260929 01:55:51.895885 28696 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:51.897105 28696 heartbeater.cc:507] Master 127.0.71.125:38709 requested a full tablet report, sending...
I20260929 01:55:51.899770 27867 ts_manager.cc:194] Re-registered known tserver with Master: a14c7d2a0afb4161926cd8cce4fbde4e (127.0.71.65:41063)
I20260929 01:55:51.902770 28695 heartbeater.cc:344] Connected to a master server at 127.0.71.126:44289
I20260929 01:55:51.902942 28695 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:51.903407 28695 heartbeater.cc:507] Master 127.0.71.126:44289 requested a full tablet report, sending...
I20260929 01:55:51.904709 28503 ts_manager.cc:194] Registered new tserver with Master: a14c7d2a0afb4161926cd8cce4fbde4e (127.0.71.65:41063)
I20260929 01:55:51.904810 28694 heartbeater.cc:344] Connected to a master server at 127.0.71.124:36311
I20260929 01:55:51.904932 28694 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:51.905452 28694 heartbeater.cc:507] Master 127.0.71.124:36311 requested a full tablet report, sending...
I20260929 01:55:51.906478 27944 ts_manager.cc:194] Re-registered known tserver with Master: a14c7d2a0afb4161926cd8cce4fbde4e (127.0.71.65:41063)
I20260929 01:55:51.907264 27944 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:55035
I20260929 01:55:52.018294 28316 heartbeater.cc:344] Connected to a master server at 127.0.71.126:44289
I20260929 01:55:52.019582 28503 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "ee03cbc827f945e58402b9cfa92b92e4" instance_seqno: 1790646949952849) as {username='kudu', principal='kudu/127.0.71.66@KRBTEST.COM'} at 127.0.71.66:53185; Asking this server to re-register.
I20260929 01:55:52.020334 28316 heartbeater.cc:461] Registering TS with master...
I20260929 01:55:52.021749 28503 ts_manager.cc:194] Registered new tserver with Master: ee03cbc827f945e58402b9cfa92b92e4 (127.0.71.66:42189)
I20260929 01:55:52.031736 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 28561
W20260929 01:55:52.033152 28708 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:55:52.024812 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:43098 (local address 127.0.71.65:41063)
0929 01:55:52.029793 (+ 4981us) server_negotiation.cc:207] Beginning negotiation
0929 01:55:52.029800 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0929 01:55:52.029828 (+ 28us) server_negotiation.cc:408] Connection header received
0929 01:55:52.029895 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:55:52.029901 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:55:52.030103 (+ 202us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:55:52.030262 (+ 159us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:55:52.031462 (+ 1200us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:43098: BlockingRecv error: recv got EOF from 127.0.0.1:43098 (error 108)
Metrics: {"server-negotiator.queue_time_us":4838,"thread_start_us":4681,"threads_started":1}
W20260929 01:55:52.034605 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 28561: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:41063: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:55:52.034737 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 28561
I20260929 01:55:52.054661 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:41063
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--webserver_port=36577
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:40297,127.0.71.85:38077,127.0.71.86:36597,127.0.71.87:37187,127.0.71.88:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:55:52.083781 28563 sys_catalog.cc:455] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "651d58de18b84e00a50539ad4f986859" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:55:52.084144 28563 sys_catalog.cc:458] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [sys.catalog]: This master's current role is: LEADER
W20260929 01:55:52.401109 28710 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:55:52.401479 28710 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:55:52.401634 28710 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:55:52.401728 28710 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:55:52.411012 28710 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:55:52.411303 28710 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:55:52.422358 28710 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:40297,127.0.71.85:38077,127.0.71.86:36597,127.0.71.87:37187,127.0.71.88:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:41063
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=36577
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:55:52.424057 28710 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:55:52.425714 28710 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:55:52.435756 28716 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:52.435918 28715 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:52.436519 28718 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:52.437809 28710 server_base.cc:1061] running on GCE node
I20260929 01:55:52.438433 28710 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:55:52.439594 28710 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:55:52.441004 28710 hybrid_clock.cc:648] HybridClock initialized: now 1790646952440835 us; error 117 us; skew 500 ppm
I20260929 01:55:52.441334 28710 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:55:52 dist-test-slave-vz23 krb5kdc[27455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:55:52.444860 28710 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:55:52.446374 28710 webserver.cc:533] Webserver started at http://127.0.71.65:36577/ using document root <none> and password file <none>
I20260929 01:55:52.447118 28710 fs_manager.cc:362] Metadata directory not provided
I20260929 01:55:52.447201 28710 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:55:52.452843 28710 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:55:52.455674 28725 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:52.457067 28710 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:55:52.457247 28710 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "a14c7d2a0afb4161926cd8cce4fbde4e"
format_stamp: "Formatted at 2026-09-29 01:55:49 on dist-test-slave-vz23"
I20260929 01:55:52.458031 28710 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:55:52.483500 28710 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:55:52.487684 28710 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:55:52.487975 28710 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:55:52.489089 28710 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:55:52.490739 28710 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:55:52.490844 28710 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:52.491000 28710 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:55:52.491071 28710 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:55:52.491444 28710 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:55:52.491582 28710 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:55:52.491674 28710 fault_injection.cc:59] injecting fault for kudu (pid 28710): FLAGS_fault_before_start (process will exit)
I20260929 01:55:52.510876 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:41063
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--webserver_port=36577
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:40297,127.0.71.85:38077,127.0.71.86:36597,127.0.71.87:37187,127.0.71.88:38849
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:55:52.857110 28733 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:55:52.857522 28733 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:55:52.857698 28733 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:55:52.866655 28733 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:55:52.866951 28733 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:55:52.878053 28733 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:40297,127.0.71.85:38077,127.0.71.86:36597,127.0.71.87:37187,127.0.71.88:38849
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:41063
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=36577
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:55:52.879781 28733 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:55:52.881243 28733 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:55:52.891407 28738 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:52.891408 28739 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:55:52.892844 28741 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:55:52.893186 28733 server_base.cc:1061] running on GCE node
I20260929 01:55:52.893806 28733 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:55:52.894976 28733 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:55:52.896181 28733 hybrid_clock.cc:648] HybridClock initialized: now 1790646952896003 us; error 148 us; skew 500 ppm
I20260929 01:55:52.896620 28733 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:55:52 dist-test-slave-vz23 krb5kdc[27455](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646952, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:55:52.899953 28733 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:55:52.901715 28733 webserver.cc:533] Webserver started at http://127.0.71.65:36577/ using document root <none> and password file <none>
I20260929 01:55:52.902446 28733 fs_manager.cc:362] Metadata directory not provided
I20260929 01:55:52.902586 28733 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:55:52.907991 28733 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:55:52.910894 28748 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:52.912333 28733 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:55:52.912550 28733 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "a14c7d2a0afb4161926cd8cce4fbde4e"
format_stamp: "Formatted at 2026-09-29 01:55:49 on dist-test-slave-vz23"
I20260929 01:55:52.913291 28733 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:55:52.946370 28733 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:55:52.950408 28733 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:55:52.950697 28733 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:55:52.951818 28733 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:55:52.953681 28733 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:55:52.953809 28733 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:52.953940 28733 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:55:52.954087 28733 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:55:52.991789 28733 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:41063
I20260929 01:55:52.991835 28860 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:41063 every 8 connection(s)
I20260929 01:55:52.993505 28733 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:55:52.995388 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 28733
I20260929 01:55:52.995823 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 28733
W20260929 01:55:53.027680 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 28733: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:41063: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:55:53.027864 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 28733
I20260929 01:55:53.044754 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 28187
W20260929 01:55:53.045316 28872 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:55:53.014885 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:42559 (local address 127.0.71.125:38709)
0929 01:55:53.025107 (+ 10222us) server_negotiation.cc:207] Beginning negotiation
0929 01:55:53.025114 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0929 01:55:53.044384 (+ 19270us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:42559: BlockingRecv error: recv got EOF from 127.0.71.65:42559 (error 108)
Metrics: {"server-negotiator.queue_time_us":10086,"thread_start_us":9988,"threads_started":1}
W20260929 01:55:53.045461 28870 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:55:53.009590 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:33307 (local address 127.0.71.126:44289)
0929 01:55:53.012829 (+ 3239us) server_negotiation.cc:207] Beginning negotiation
0929 01:55:53.012839 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0929 01:55:53.022284 (+ 9445us) server_negotiation.cc:408] Connection header received
0929 01:55:53.024248 (+ 1964us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:55:53.024256 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:55:53.024353 (+ 97us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:55:53.024503 (+ 150us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:55:53.044566 (+ 20063us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:33307: BlockingRecv error: recv got EOF from 127.0.71.65:33307 (error 108)
Metrics: {"server-negotiator.queue_time_us":3088,"thread_start_us":2988,"threads_started":1}
W20260929 01:55:53.046238 28866 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:55:52.997188 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:56187 (local address 127.0.71.124:36311)
0929 01:55:53.024572 (+ 27384us) server_negotiation.cc:207] Beginning negotiation
0929 01:55:53.024579 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0929 01:55:53.024598 (+ 19us) server_negotiation.cc:408] Connection header received
0929 01:55:53.024659 (+ 61us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:55:53.024665 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:55:53.024764 (+ 99us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:55:53.024900 (+ 136us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:55:53.045277 (+ 20377us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:56187: BlockingRecv error: recv got EOF from 127.0.71.65:56187 (error 108)
Metrics: {"server-negotiator.queue_time_us":27231,"thread_start_us":28272,"threads_started":1}
W20260929 01:55:53.047716 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 28187: Not authorized: Client connection negotiation failed: client connection to 127.0.71.66:42189: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:55:53.047899 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 28187
I20260929 01:55:53.062817 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 28329
W20260929 01:55:53.065982 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 28329: Not authorized: Client connection negotiation failed: client connection to 127.0.71.67:37973: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:55:53.066138 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 28329
I20260929 01:55:53.078867 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 28468
W20260929 01:55:53.081386 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 28468: Not authorized: Client connection negotiation failed: client connection to 127.0.71.126:44289: server requires authentication, but client does not have Kerberos credentials available
W20260929 01:55:53.081503 28870 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:55:53.079912 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:40720 (local address 127.0.71.126:44289)
0929 01:55:53.080090 (+ 178us) server_negotiation.cc:207] Beginning negotiation
0929 01:55:53.080097 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0929 01:55:53.080146 (+ 49us) server_negotiation.cc:408] Connection header received
0929 01:55:53.080384 (+ 238us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:55:53.080390 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:55:53.080518 (+ 128us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:55:53.080632 (+ 114us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:55:53.081310 (+ 678us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40720: BlockingRecv error: recv got EOF from 127.0.0.1:40720 (error 108)
Metrics: {"server-negotiator.queue_time_us":66}
I20260929 01:55:53.081586 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 28468
I20260929 01:55:53.101653 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 27835
I20260929 01:55:53.102897 27897 generic_service.cc:196] Checking for leaks (request via RPC)
W20260929 01:55:53.114348 27929 consensus_peers.cc:597] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 -> Peer e128f9c5227a486482933c152890ec12 (127.0.71.126:44289): Couldn't send request to peer e128f9c5227a486482933c152890ec12. Status: Network error: Client connection negotiation failed: client connection to 127.0.71.126:44289: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 01:55:53.164500 27912 master_runner.cc:187] Error getting master registration for 127.0.71.126:44289: Network error: Client connection negotiation failed: client connection to 127.0.71.126:44289: connect: Connection refused (error 111),
I20260929 01:55:53.164757 27912 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:55:53.215009 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 27835
I20260929 01:55:53.235975 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 27912
I20260929 01:55:53.237296 27974 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:55:53.352025 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 27912
Shutting down hive metastore.
2026-09-29T01:55:53Z chronyd exiting
2026-09-29T01:55:53Z chronyd exiting
2026-09-29T01:55:53Z chronyd exiting
2026-09-29T01:55:53Z chronyd exiting
2026-09-29T01:55:53Z chronyd exiting
Sep 29 01:55:53 dist-test-slave-vz23 krb5kdc[28884](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:55:53 dist-test-slave-vz23 krb5kdc[28884](info): set up 2 sockets
Sep 29 01:55:53 dist-test-slave-vz23 krb5kdc[28884](info): commencing operation
krb5kdc: starting...
W20260929 01:55:55.837124 285 mini_kdc.cc:121] Time spent starting KDC: real 2.043s user 0.005s sys 0.013s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-admin@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "test-user@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "joe-interloper@KRBTEST.COM".
Authenticating as principal slave/admin@KRBTEST.COM with password.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 01:55:55 dist-test-slave-vz23 krb5kdc[28884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:55:55Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:55:55Z Disabled control of system clock
2026-09-29T01:55:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:55:56Z Disabled control of system clock
2026-09-29T01:55:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:55:56Z Disabled control of system clock
2026-09-29T01:55:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:55:56Z Disabled control of system clock
2026-09-29T01:55:56Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:55:56Z Disabled control of system clock
WARNING: no policy specified for hive/127.0.0.1@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "hive/127.0.0.1@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal hive/127.0.0.1 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/hive_127.0.0.1.keytab.
Entry for principal hive/127.0.0.1 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/hive_127.0.0.1.keytab.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5.conf
2026-09-29 01:55:57: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
Sep 29 01:55:59 dist-test-slave-vz23 krb5kdc[28884](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790646959, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 01:56:04 dist-test-slave-vz23 krb5kdc[28884](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790646964, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 01:56:04 dist-test-slave-vz23 krb5kdc[28884](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790646964, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.126@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.126@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:56:05.680197 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:44289
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:37059,127.0.71.85:38555,127.0.71.86:35797,127.0.71.87:35253,127.0.71.88:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:56:06.018469 29086 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:06.018790 29086 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:06.018944 29086 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:06.027287 29086 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:56:06.027400 29086 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:06.027518 29086 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:56:06.027616 29086 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:56:06.038662 29086 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:37059,127.0.71.85:38555,127.0.71.86:35797,127.0.71.87:35253,127.0.71.88:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:44289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:06.040308 29086 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:06.041745 29086 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:06.050979 29091 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:06.051010 29092 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:06.051196 29086 server_base.cc:1061] running on GCE node
W20260929 01:56:06.050932 29094 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:06.052395 29086 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:06.054199 29086 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:06.055493 29086 hybrid_clock.cc:648] HybridClock initialized: now 1790646966055357 us; error 107 us; skew 500 ppm
I20260929 01:56:06.055877 29086 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:56:06 dist-test-slave-vz23 krb5kdc[28884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:56:06.059561 29086 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:56:06.061528 29086 webserver.cc:533] Webserver started at http://127.0.71.126:46339/ using document root <none> and password file <none>
I20260929 01:56:06.062281 29086 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:06.062381 29086 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:06.068110 29086 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 01:56:06.071136 29101 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:06.072657 29086 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:56:06.072845 29086 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "e128f9c5227a486482933c152890ec12"
format_stamp: "Formatted at 2026-09-29 01:55:46 on dist-test-slave-vz23"
I20260929 01:56:06.073700 29086 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:06.090528 29086 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:06.101567 29086 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:06.101861 29086 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:06.125893 29086 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:44289
I20260929 01:56:06.125898 29152 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:44289 every 8 connection(s)
I20260929 01:56:06.127794 29086 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:56:06.134677 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 29086
I20260929 01:56:06.134917 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:56:06.134990 29153 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:56:06.138348 29153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: Bootstrap starting.
I20260929 01:56:06.154989 29153 log.cc:826] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: Log is configured to *not* fsync() on all Append() calls
WARNING: no policy specified for kudu/127.0.71.125@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.125@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:56:06.163616 29153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:56:06.164134 29153 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: Bootstrap complete.
I20260929 01:56:06.170930 29153 raft_consensus.cc:359] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } }
I20260929 01:56:06.171635 29153 raft_consensus.cc:740] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e128f9c5227a486482933c152890ec12, State: Initialized, Role: FOLLOWER
I20260929 01:56:06.172493 29153 consensus_queue.cc:260] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } }
I20260929 01:56:06.173413 29158 sys_catalog.cc:455] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:56:06.173717 29158 sys_catalog.cc:458] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:06.174526 29153 sys_catalog.cc:565] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [sys.catalog]: configured and running, proceeding with master startup.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.125 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.125 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:56:06.185858 29169 catalog_manager.cc:1308] Loaded cluster ID: 63a1f8021ce442519704c5b15912a2a3
I20260929 01:56:06.185985 29169 catalog_manager.cc:1601] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: loading cluster ID for follower catalog manager: success
I20260929 01:56:06.191443 29169 catalog_manager.cc:1623] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: acquiring CA information for follower catalog manager: success
I20260929 01:56:06.194036 29169 catalog_manager.cc:1651] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for HTTP/127.0.71.125@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.125@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:56:06.211560 29086 master_runner.cc:187] Error getting master registration for 127.0.71.125:38709: Network error: Client connection negotiation failed: client connection to 127.0.71.125:38709: connect: Connection refused (error 111),
I20260929 01:56:06.215903 29086 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.125 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.125 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:56:06.245621 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:38709
--webserver_interface=127.0.71.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:37059,127.0.71.85:38555,127.0.71.86:35797,127.0.71.87:35253,127.0.71.88:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:56:06.609072 29175 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:06.609465 29175 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:06.609613 29175 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:06.618216 29175 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:56:06.618371 29175 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:06.618486 29175 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:56:06.618577 29175 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:56:06.629801 29175 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:37059,127.0.71.85:38555,127.0.71.86:35797,127.0.71.87:35253,127.0.71.88:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:38709
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:06.631456 29175 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:06.632943 29175 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:06.641353 29181 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:06.641670 29180 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:06.642781 29183 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:06.643589 29175 server_base.cc:1061] running on GCE node
I20260929 01:56:06.644548 29175 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:06.646209 29175 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:06.647480 29175 hybrid_clock.cc:648] HybridClock initialized: now 1790646966647355 us; error 92 us; skew 500 ppm
I20260929 01:56:06.647897 29175 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:56:06 dist-test-slave-vz23 krb5kdc[28884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:56:06.651692 29175 init.cc:377] Logged in from keytab as kudu/127.0.71.125@KRBTEST.COM (short username kudu)
I20260929 01:56:06.653618 29175 webserver.cc:533] Webserver started at http://127.0.71.125:40985/ using document root <none> and password file <none>
I20260929 01:56:06.654356 29175 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:06.654508 29175 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:06.660225 29175 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.000s
I20260929 01:56:06.663599 29190 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:06.665175 29175 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:56:06.665381 29175 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "d0620fc4eb154a439efacce291731d63"
format_stamp: "Formatted at 2026-09-29 01:55:47 on dist-test-slave-vz23"
I20260929 01:56:06.666234 29175 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:06.714349 29175 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:06.719058 29175 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:06.719367 29175 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:06.742838 29175 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:38709
I20260929 01:56:06.742874 29241 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:38709 every 8 connection(s)
I20260929 01:56:06.744705 29175 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:56:06.752271 29242 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:56:06.754863 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 29175
I20260929 01:56:06.755074 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:56:06.756366 29242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63: Bootstrap starting.
I20260929 01:56:06.777295 29242 log.cc:826] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63: Log is configured to *not* fsync() on all Append() calls
WARNING: no policy specified for kudu/127.0.71.124@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.124@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:56:06.788976 29242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:56:06.789675 29242 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63: Bootstrap complete.
I20260929 01:56:06.797679 29242 raft_consensus.cc:359] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } }
I20260929 01:56:06.798448 29242 raft_consensus.cc:740] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: d0620fc4eb154a439efacce291731d63, State: Initialized, Role: FOLLOWER
I20260929 01:56:06.799225 29242 consensus_queue.cc:260] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } }
I20260929 01:56:06.800199 29247 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:56:06.800585 29247 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:06.801596 29242 sys_catalog.cc:565] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [sys.catalog]: configured and running, proceeding with master startup.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.124 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.124 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:56:06.815485 29258 catalog_manager.cc:1308] Loaded cluster ID: 63a1f8021ce442519704c5b15912a2a3
I20260929 01:56:06.815587 29258 catalog_manager.cc:1601] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63: loading cluster ID for follower catalog manager: success
I20260929 01:56:06.819561 29258 catalog_manager.cc:1623] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63: acquiring CA information for follower catalog manager: success
I20260929 01:56:06.822962 29258 catalog_manager.cc:1651] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Sep 29 01:56:06 dist-test-slave-vz23 krb5kdc[28884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
WARNING: no policy specified for HTTP/127.0.71.124@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.124@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:56:06.853492 29175 master_runner.cc:187] Error getting master registration for 127.0.71.124:36311: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36311: connect: Connection refused (error 111),
I20260929 01:56:06.859315 29175 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.124 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.124 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:56:06.876852 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:36311
--webserver_interface=127.0.71.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:37059,127.0.71.85:38555,127.0.71.86:35797,127.0.71.87:35253,127.0.71.88:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:56:07.225524 29265 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:07.225839 29265 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:07.225921 29265 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:07.234298 29265 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:56:07.234407 29265 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:07.234462 29265 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:56:07.234517 29265 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:56:07.245818 29265 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:37059,127.0.71.85:38555,127.0.71.86:35797,127.0.71.87:35253,127.0.71.88:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:36311
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:07.247356 29265 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:07.248723 29265 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:07.257194 29271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:07.257149 29270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:07.257653 29273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:07.258630 29265 server_base.cc:1061] running on GCE node
I20260929 01:56:07.259342 29265 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:07.260990 29265 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:07.262224 29265 hybrid_clock.cc:648] HybridClock initialized: now 1790646967262055 us; error 136 us; skew 500 ppm
I20260929 01:56:07.262612 29265 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:56:07 dist-test-slave-vz23 krb5kdc[28884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:56:07.266172 29265 init.cc:377] Logged in from keytab as kudu/127.0.71.124@KRBTEST.COM (short username kudu)
I20260929 01:56:07.267850 29265 webserver.cc:533] Webserver started at http://127.0.71.124:42229/ using document root <none> and password file <none>
I20260929 01:56:07.268636 29265 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:07.268780 29265 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:07.274372 29265 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:56:07.277437 29280 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:07.279045 29265 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:56:07.279222 29265 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "651d58de18b84e00a50539ad4f986859"
format_stamp: "Formatted at 2026-09-29 01:55:47 on dist-test-slave-vz23"
I20260929 01:56:07.279935 29265 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:07.300802 29265 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:07.305115 29265 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:07.305414 29265 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:07.325767 29328 raft_consensus.cc:493] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:56:07.326073 29328 raft_consensus.cc:515] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } }
I20260929 01:56:07.327124 29265 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:36311
I20260929 01:56:07.327206 29332 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:36311 every 8 connection(s)
I20260929 01:56:07.328364 29328 leader_election.cc:290] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers d0620fc4eb154a439efacce291731d63 (127.0.71.125:38709), 651d58de18b84e00a50539ad4f986859 (127.0.71.124:36311)
I20260929 01:56:07.329054 29265 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:56:07.331729 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 29265
I20260929 01:56:07.331969 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:56:07.341928 29335 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:56:07.347291 29335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859: Bootstrap starting.
I20260929 01:56:07.353557 29217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e128f9c5227a486482933c152890ec12" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0620fc4eb154a439efacce291731d63" is_pre_election: true
I20260929 01:56:07.354183 29217 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e128f9c5227a486482933c152890ec12 in term 1.
I20260929 01:56:07.355015 29104 leader_election.cc:304] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0620fc4eb154a439efacce291731d63, e128f9c5227a486482933c152890ec12; no voters:
I20260929 01:56:07.355518 29328 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 01:56:07.355681 29328 raft_consensus.cc:493] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:56:07.355819 29328 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [term 1 FOLLOWER]: Advancing to term 2
Sep 29 01:56:07 dist-test-slave-vz23 krb5kdc[28884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646966, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:56:07.362274 29328 raft_consensus.cc:515] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } }
I20260929 01:56:07.363780 29217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e128f9c5227a486482933c152890ec12" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "d0620fc4eb154a439efacce291731d63"
I20260929 01:56:07.364040 29217 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:56:07.369628 29335 log.cc:826] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859: Log is configured to *not* fsync() on all Append() calls
I20260929 01:56:07.372218 29217 raft_consensus.cc:2468] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e128f9c5227a486482933c152890ec12 in term 2.
I20260929 01:56:07.372780 29104 leader_election.cc:304] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0620fc4eb154a439efacce291731d63, e128f9c5227a486482933c152890ec12; no voters:
I20260929 01:56:07.373819 29341 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:56:07.374305 29328 leader_election.cc:290] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [CANDIDATE]: Term 2 election: Requested vote from peers d0620fc4eb154a439efacce291731d63 (127.0.71.125:38709), 651d58de18b84e00a50539ad4f986859 (127.0.71.124:36311)
I20260929 01:56:07.378518 29341 raft_consensus.cc:697] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [term 2 LEADER]: Becoming Leader. State: Replica: e128f9c5227a486482933c152890ec12, State: Running, Role: LEADER
I20260929 01:56:07.379537 29341 consensus_queue.cc:237] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } }
I20260929 01:56:07.383036 29335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859: Bootstrap replayed 1/1 log segments. Stats: ops{read=4 overwritten=0 applied=4 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:56:07.382963 29328 sys_catalog.cc:455] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e128f9c5227a486482933c152890ec12. Latest consensus state: current_term: 2 leader_uuid: "e128f9c5227a486482933c152890ec12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:56:07.383208 29328 sys_catalog.cc:458] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:07.383759 29335 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859: Bootstrap complete.
I20260929 01:56:07.384207 29344 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:56:07.387408 29344 catalog_manager.cc:1533] Initializing Kudu cluster ID...
Sep 29 01:56:07 dist-test-slave-vz23 krb5kdc[28884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:56:07.388363 29344 catalog_manager.cc:1308] Loaded cluster ID: 63a1f8021ce442519704c5b15912a2a3
I20260929 01:56:07.388477 29344 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:56:07.390106 29344 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:56:07.391844 29344 catalog_manager.cc:6230] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: Loaded TSK: 0
I20260929 01:56:07.393036 29344 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:56:07.392992 29335 raft_consensus.cc:359] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } }
I20260929 01:56:07.393986 29335 raft_consensus.cc:740] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 651d58de18b84e00a50539ad4f986859, State: Initialized, Role: FOLLOWER
I20260929 01:56:07.394776 29335 consensus_queue.cc:260] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 4, Last appended: 1.4, Last appended by leader: 4, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } }
I20260929 01:56:07.395762 29345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:56:07.396085 29345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:07.397173 29335 sys_catalog.cc:565] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:56:07.397228 29307 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e128f9c5227a486482933c152890ec12" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "651d58de18b84e00a50539ad4f986859" is_pre_election: true
I20260929 01:56:07.397329 29306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e128f9c5227a486482933c152890ec12" candidate_term: 2 candidate_status { last_received { term: 1 index: 4 } } ignore_live_leader: false dest_uuid: "651d58de18b84e00a50539ad4f986859"
I20260929 01:56:07.397833 29306 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:56:07.405917 29306 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e128f9c5227a486482933c152890ec12 in term 2.
Sep 29 01:56:07 dist-test-slave-vz23 krb5kdc[28884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:56:07.425514 29356 catalog_manager.cc:1308] Loaded cluster ID: 63a1f8021ce442519704c5b15912a2a3
I20260929 01:56:07.425705 29356 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859: loading cluster ID for follower catalog manager: success
Sep 29 01:56:07 dist-test-slave-vz23 krb5kdc[28884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:56:07.441398 29356 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859: acquiring CA information for follower catalog manager: success
Sep 29 01:56:07 dist-test-slave-vz23 krb5kdc[28884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.65@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:56:07.449316 29356 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.65@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:56:07.545161 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:37059,127.0.71.85:38555,127.0.71.86:35797,127.0.71.87:35253,127.0.71.88:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:56:07.826249 29306 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [term 2 FOLLOWER]: Refusing update from remote peer e128f9c5227a486482933c152890ec12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 01:56:07.827255 29341 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:56:07.831727 29345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e128f9c5227a486482933c152890ec12. Latest consensus state: current_term: 2 leader_uuid: "e128f9c5227a486482933c152890ec12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:56:07.831992 29345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:07.845408 29345 sys_catalog.cc:455] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "e128f9c5227a486482933c152890ec12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:56:07.845649 29345 sys_catalog.cc:458] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:07.847909 29341 sys_catalog.cc:455] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "e128f9c5227a486482933c152890ec12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:56:07.848203 29341 sys_catalog.cc:458] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:07.849495 29217 raft_consensus.cc:1275] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [term 2 FOLLOWER]: Refusing update from remote peer e128f9c5227a486482933c152890ec12: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 2 index: 5. (index mismatch)
I20260929 01:56:07.850502 29328 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [LEADER]: Connected to new peer: Peer: permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:56:07.858598 29370 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "e128f9c5227a486482933c152890ec12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:56:07.858961 29370 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:07.859458 29367 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e128f9c5227a486482933c152890ec12. Latest consensus state: current_term: 2 leader_uuid: "e128f9c5227a486482933c152890ec12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:56:07.859795 29367 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:56:07.916376 29363 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:07.916780 29363 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:07.916950 29363 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:07.925840 29363 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:07.926095 29363 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:56:07.937261 29363 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:37059,127.0.71.85:38555,127.0.71.86:35797,127.0.71.87:35253,127.0.71.88:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:07.938956 29363 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:07.940342 29363 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:07.950551 29375 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:07.950533 29376 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:07.951192 29378 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:07.952030 29363 server_base.cc:1061] running on GCE node
I20260929 01:56:07.952764 29363 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:07.953926 29363 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:07.955236 29363 hybrid_clock.cc:648] HybridClock initialized: now 1790646967955068 us; error 139 us; skew 500 ppm
I20260929 01:56:07.955583 29363 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:56:07 dist-test-slave-vz23 krb5kdc[28884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:56:07.959409 29363 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:56:07.961225 29363 webserver.cc:533] Webserver started at http://127.0.71.65:34139/ using document root <none> and password file <none>
I20260929 01:56:07.962018 29363 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:07.962185 29363 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:07.968523 29363 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.006s sys 0.000s
I20260929 01:56:07.971797 29385 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:07.973588 29363 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:56:07.973794 29363 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "a14c7d2a0afb4161926cd8cce4fbde4e"
format_stamp: "Formatted at 2026-09-29 01:55:49 on dist-test-slave-vz23"
I20260929 01:56:07.974615 29363 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:08.001224 29363 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:08.005892 29363 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:08.006134 29363 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:08.007128 29363 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:56:08.008740 29363 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:56:08.008836 29363 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:08.008917 29363 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:56:08.008977 29363 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:08.046207 29363 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:37331
I20260929 01:56:08.046360 29497 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:37331 every 8 connection(s)
I20260929 01:56:08.047919 29363 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:56:08.055078 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 29363
I20260929 01:56:08.055306 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
Sep 29 01:56:08 dist-test-slave-vz23 krb5kdc[28884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:56:08 dist-test-slave-vz23 krb5kdc[28884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:56:08 dist-test-slave-vz23 krb5kdc[28884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646967, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:56:08.080828 29499 heartbeater.cc:344] Connected to a master server at 127.0.71.126:44289
I20260929 01:56:08.081336 29499 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:08.082602 29499 heartbeater.cc:507] Master 127.0.71.126:44289 requested a full tablet report, sending...
I20260929 01:56:08.083667 29498 heartbeater.cc:344] Connected to a master server at 127.0.71.124:36311
I20260929 01:56:08.083840 29498 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:08.084435 29498 heartbeater.cc:507] Master 127.0.71.124:36311 requested a full tablet report, sending...
I20260929 01:56:08.085906 29118 ts_manager.cc:194] Registered new tserver with Master: a14c7d2a0afb4161926cd8cce4fbde4e (127.0.71.65:37331)
I20260929 01:56:08.087738 29118 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:50099
I20260929 01:56:08.090538 29297 ts_manager.cc:194] Registered new tserver with Master: a14c7d2a0afb4161926cd8cce4fbde4e (127.0.71.65:37331)
I20260929 01:56:08.091645 29500 heartbeater.cc:344] Connected to a master server at 127.0.71.125:38709
I20260929 01:56:08.091831 29500 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:08.092272 29500 heartbeater.cc:507] Master 127.0.71.125:38709 requested a full tablet report, sending...
WARNING: no policy specified for kudu/127.0.71.66@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.66@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
I20260929 01:56:08.094422 29207 ts_manager.cc:194] Registered new tserver with Master: a14c7d2a0afb4161926cd8cce4fbde4e (127.0.71.65:37331)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.66 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.66 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.66@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.66@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.66 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.66 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:56:08.180931 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.66:0
--local_ip_for_outbound_sockets=127.0.71.66
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.66
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:37059,127.0.71.85:38555,127.0.71.86:35797,127.0.71.87:35253,127.0.71.88:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:56:08.272578 29342 sys_catalog.cc:455] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "e128f9c5227a486482933c152890ec12" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:56:08.272917 29342 sys_catalog.cc:458] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [sys.catalog]: This master's current role is: LEADER
W20260929 01:56:08.543094 29512 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:08.543433 29512 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:08.543618 29512 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:08.552884 29512 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:08.553139 29512 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.66
I20260929 01:56:08.564396 29512 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:37059,127.0.71.85:38555,127.0.71.86:35797,127.0.71.87:35253,127.0.71.88:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.66
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.66:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.71.66
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:08.566195 29512 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:08.567690 29512 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:08.577481 29517 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:08.578269 29518 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:08.578841 29520 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:08.580183 29512 server_base.cc:1061] running on GCE node
I20260929 01:56:08.581002 29512 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:08.582369 29512 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:08.583765 29512 hybrid_clock.cc:648] HybridClock initialized: now 1790646968583589 us; error 143 us; skew 500 ppm
I20260929 01:56:08.584082 29512 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:56:08 dist-test-slave-vz23 krb5kdc[28884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:56:08.588697 29512 init.cc:377] Logged in from keytab as kudu/127.0.71.66@KRBTEST.COM (short username kudu)
I20260929 01:56:08.590827 29512 webserver.cc:533] Webserver started at http://127.0.71.66:34083/ using document root <none> and password file <none>
I20260929 01:56:08.591636 29512 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:08.591776 29512 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:08.597637 29512 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:56:08.600863 29527 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:08.602337 29512 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:56:08.602550 29512 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/wal
uuid: "ee03cbc827f945e58402b9cfa92b92e4"
format_stamp: "Formatted at 2026-09-29 01:55:49 on dist-test-slave-vz23"
I20260929 01:56:08.603338 29512 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:08.632381 29512 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:08.637043 29512 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:08.637382 29512 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:08.638394 29512 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:56:08.640026 29512 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:56:08.640115 29512 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:08.640275 29512 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:56:08.640354 29512 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:08.676376 29512 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.66:45059
I20260929 01:56:08.676621 29639 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.66:45059 every 8 connection(s)
I20260929 01:56:08.678056 29512 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
I20260929 01:56:08.678448 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 29512
I20260929 01:56:08.678690 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-1/wal/instance
WARNING: no policy specified for kudu/127.0.71.67@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "kudu/127.0.71.67@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Sep 29 01:56:08 dist-test-slave-vz23 krb5kdc[28884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:56:08 dist-test-slave-vz23 krb5kdc[28884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:56:08 dist-test-slave-vz23 krb5kdc[28884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646968, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.66@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:56:08.731238 29641 heartbeater.cc:344] Connected to a master server at 127.0.71.126:44289
I20260929 01:56:08.731653 29641 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:08.732734 29641 heartbeater.cc:507] Master 127.0.71.126:44289 requested a full tablet report, sending...
I20260929 01:56:08.733032 29642 heartbeater.cc:344] Connected to a master server at 127.0.71.125:38709
I20260929 01:56:08.733199 29642 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:08.733747 29642 heartbeater.cc:507] Master 127.0.71.125:38709 requested a full tablet report, sending...
I20260929 01:56:08.734689 29118 ts_manager.cc:194] Registered new tserver with Master: ee03cbc827f945e58402b9cfa92b92e4 (127.0.71.66:45059)
I20260929 01:56:08.734987 29207 ts_manager.cc:194] Registered new tserver with Master: ee03cbc827f945e58402b9cfa92b92e4 (127.0.71.66:45059)
I20260929 01:56:08.735857 29118 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.66@KRBTEST.COM'} at 127.0.71.66:38599
I20260929 01:56:08.739436 29640 heartbeater.cc:344] Connected to a master server at 127.0.71.124:36311
I20260929 01:56:08.739588 29640 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:08.739950 29640 heartbeater.cc:507] Master 127.0.71.124:36311 requested a full tablet report, sending...
I20260929 01:56:08.741178 29297 ts_manager.cc:194] Registered new tserver with Master: ee03cbc827f945e58402b9cfa92b92e4 (127.0.71.66:45059)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.67 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.67 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.67@KRBTEST.COM; defaulting to no policy
add_principal: Principal or policy already exists while creating "HTTP/127.0.71.67@KRBTEST.COM".
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.67 with kvno 3, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.67 with kvno 3, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:56:08.813540 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.67:0
--local_ip_for_outbound_sockets=127.0.71.67
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.67
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:37059,127.0.71.85:38555,127.0.71.86:35797,127.0.71.87:35253,127.0.71.88:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:56:09.091913 29499 heartbeater.cc:499] Master 127.0.71.126:44289 was elected leader, sending a full tablet report...
W20260929 01:56:09.196506 29654 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:09.196864 29654 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:09.196977 29654 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:09.206245 29654 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:09.206475 29654 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.67
I20260929 01:56:09.218269 29654 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:37059,127.0.71.85:38555,127.0.71.86:35797,127.0.71.87:35253,127.0.71.88:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.67
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.67:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.71.67
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:09.219885 29654 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:09.221377 29654 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:09.231968 29659 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:09.232862 29660 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:09.233278 29662 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:09.234086 29654 server_base.cc:1061] running on GCE node
I20260929 01:56:09.234736 29654 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:09.235900 29654 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:09.237257 29654 hybrid_clock.cc:648] HybridClock initialized: now 1790646969237102 us; error 119 us; skew 500 ppm
I20260929 01:56:09.237692 29654 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:56:09 dist-test-slave-vz23 krb5kdc[28884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:56:09.242184 29654 init.cc:377] Logged in from keytab as kudu/127.0.71.67@KRBTEST.COM (short username kudu)
I20260929 01:56:09.244117 29654 webserver.cc:533] Webserver started at http://127.0.71.67:36735/ using document root <none> and password file <none>
I20260929 01:56:09.244997 29654 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:09.245155 29654 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:09.251226 29654 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:56:09.254108 29669 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:09.255661 29654 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:56:09.255878 29654 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/wal
uuid: "bc0338288524439284a8f6b010ef98c1"
format_stamp: "Formatted at 2026-09-29 01:55:50 on dist-test-slave-vz23"
I20260929 01:56:09.256669 29654 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:09.278549 29654 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:09.283010 29654 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:09.283324 29654 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:09.284339 29654 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:56:09.286062 29654 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:56:09.286192 29654 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:09.286366 29654 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:56:09.286475 29654 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:09.324370 29654 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.67:43509
I20260929 01:56:09.324426 29781 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.67:43509 every 8 connection(s)
I20260929 01:56:09.325925 29654 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
I20260929 01:56:09.331602 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 29654
I20260929 01:56:09.331844 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-2/wal/instance
Sep 29 01:56:09 dist-test-slave-vz23 krb5kdc[28884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:56:09 dist-test-slave-vz23 krb5kdc[28884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:56:09 dist-test-slave-vz23 krb5kdc[28884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.67@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:56:09.369225 29783 heartbeater.cc:344] Connected to a master server at 127.0.71.126:44289
I20260929 01:56:09.369343 29784 heartbeater.cc:344] Connected to a master server at 127.0.71.125:38709
I20260929 01:56:09.369650 29783 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:09.369655 29784 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:09.370131 29782 heartbeater.cc:344] Connected to a master server at 127.0.71.124:36311
I20260929 01:56:09.370280 29782 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:09.370820 29784 heartbeater.cc:507] Master 127.0.71.125:38709 requested a full tablet report, sending...
I20260929 01:56:09.370846 29783 heartbeater.cc:507] Master 127.0.71.126:44289 requested a full tablet report, sending...
I20260929 01:56:09.370838 29782 heartbeater.cc:507] Master 127.0.71.124:36311 requested a full tablet report, sending...
I20260929 01:56:09.373370 29118 ts_manager.cc:194] Registered new tserver with Master: bc0338288524439284a8f6b010ef98c1 (127.0.71.67:43509)
I20260929 01:56:09.373613 29207 ts_manager.cc:194] Registered new tserver with Master: bc0338288524439284a8f6b010ef98c1 (127.0.71.67:43509)
I20260929 01:56:09.373660 29297 ts_manager.cc:194] Registered new tserver with Master: bc0338288524439284a8f6b010ef98c1 (127.0.71.67:43509)
I20260929 01:56:09.374573 29118 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.67@KRBTEST.COM'} at 127.0.71.67:46819
I20260929 01:56:09.386184 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:56:09.396283 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 01:56:09.396615 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 29086
I20260929 01:56:09.397753 29148 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:09.515306 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 29086
I20260929 01:56:09.538405 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:44289
--webserver_interface=127.0.71.126
--webserver_port=46339
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:37059,127.0.71.85:38555,127.0.71.86:35797,127.0.71.87:35253,127.0.71.88:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:56:09.738850 29641 heartbeater.cc:499] Master 127.0.71.126:44289 was elected leader, sending a full tablet report...
W20260929 01:56:09.740489 29641 heartbeater.cc:646] Failed to heartbeat to 127.0.71.126:44289 (0 consecutive failures): Network error: Failed to send heartbeat to master: Client connection negotiation failed: client connection to 127.0.71.126:44289: connect: Connection refused (error 111)
W20260929 01:56:09.906963 29793 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:09.907341 29793 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:09.907505 29793 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:09.918069 29793 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:56:09.918191 29793 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:09.918308 29793 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:56:09.918411 29793 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:56:09.931195 29793 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:37059,127.0.71.85:38555,127.0.71.86:35797,127.0.71.87:35253,127.0.71.88:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:44289
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=46339
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:09.932981 29793 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:09.934477 29793 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:09.946064 29799 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:09.946062 29800 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:09.946903 29802 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:09.947438 29793 server_base.cc:1061] running on GCE node
I20260929 01:56:09.948261 29793 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:09.949981 29793 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:09.951223 29793 hybrid_clock.cc:648] HybridClock initialized: now 1790646969951087 us; error 105 us; skew 500 ppm
I20260929 01:56:09.951584 29793 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:56:09 dist-test-slave-vz23 krb5kdc[28884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:56:09.955294 29793 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:56:09.956887 29793 webserver.cc:533] Webserver started at http://127.0.71.126:46339/ using document root <none> and password file <none>
I20260929 01:56:09.957552 29793 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:09.957640 29793 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:09.963121 29793 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:56:09.966120 29809 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:09.967495 29793 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:56:09.967631 29793 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "e128f9c5227a486482933c152890ec12"
format_stamp: "Formatted at 2026-09-29 01:55:46 on dist-test-slave-vz23"
I20260929 01:56:09.968350 29793 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:09.995465 29793 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:10.000087 29793 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:10.000334 29793 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:10.023264 29793 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:44289
I20260929 01:56:10.023418 29860 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:44289 every 8 connection(s)
I20260929 01:56:10.024973 29793 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:56:10.026109 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 29793
I20260929 01:56:10.026441 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 29363
I20260929 01:56:10.035104 29861 sys_catalog.cc:263] Verifying existing consensus state
Sep 29 01:56:10 dist-test-slave-vz23 krb5kdc[28884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 01:56:10.040684 29861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: Bootstrap starting.
I20260929 01:56:10.049311 29493 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:10.065069 29861 log.cc:826] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: Log is configured to *not* fsync() on all Append() calls
I20260929 01:56:10.074645 29861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:56:10.075232 29861 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: Bootstrap complete.
I20260929 01:56:10.082468 29861 raft_consensus.cc:359] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [term 2 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } }
I20260929 01:56:10.083236 29861 raft_consensus.cc:740] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [term 2 FOLLOWER]: Becoming Follower/Learner. State: Replica: e128f9c5227a486482933c152890ec12, State: Initialized, Role: FOLLOWER
I20260929 01:56:10.083985 29861 consensus_queue.cc:260] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } }
I20260929 01:56:10.084971 29867 sys_catalog.cc:455] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 2 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:56:10.085299 29867 sys_catalog.cc:458] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:10.086197 29861 sys_catalog.cc:565] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:56:10.096894 29878 catalog_manager.cc:1308] Loaded cluster ID: 63a1f8021ce442519704c5b15912a2a3
I20260929 01:56:10.103372 29878 catalog_manager.cc:1601] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: loading cluster ID for follower catalog manager: success
I20260929 01:56:10.108215 29878 catalog_manager.cc:1623] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: acquiring CA information for follower catalog manager: success
I20260929 01:56:10.113209 29878 catalog_manager.cc:1651] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Sep 29 01:56:10 dist-test-slave-vz23 krb5kdc[28884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646969, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:56:10.153824 29793 master_runner.cc:264] No leader master found from master e128f9c5227a486482933c152890ec12
I20260929 01:56:10.153988 29793 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:56:10.181591 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 29363
W20260929 01:56:10.198671 29283 connection.cc:573] server connection from 127.0.71.65:53639 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:56:10.199700 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:37331
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--webserver_port=34139
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:37059,127.0.71.85:38555,127.0.71.86:35797,127.0.71.87:35253,127.0.71.88:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:56:10.200026 29880 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:56:10.180380 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:34675 (local address 127.0.71.126:44289)
0929 01:56:10.183625 (+ 3245us) server_negotiation.cc:207] Beginning negotiation
0929 01:56:10.183629 (+ 4us) server_negotiation.cc:400] Waiting for connection header
0929 01:56:10.198740 (+ 15111us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:34675: BlockingRecv error: recv got EOF from 127.0.71.65:34675 (error 108)
Metrics: {"server-negotiator.queue_time_us":3122}
I20260929 01:56:10.377322 29783 heartbeater.cc:499] Master 127.0.71.126:44289 was elected leader, sending a full tablet report...
I20260929 01:56:10.393867 29826 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "bc0338288524439284a8f6b010ef98c1" instance_seqno: 1790646969315153) as {username='kudu', principal='kudu/127.0.71.67@KRBTEST.COM'} at 127.0.71.67:59145; Asking this server to re-register.
I20260929 01:56:10.395292 29783 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:10.397641 29826 ts_manager.cc:194] Registered new tserver with Master: bc0338288524439284a8f6b010ef98c1 (127.0.71.67:43509)
W20260929 01:56:10.588912 29885 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:10.589257 29885 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:10.589414 29885 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:10.598272 29885 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:10.598536 29885 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:56:10.609807 29885 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:37059,127.0.71.85:38555,127.0.71.86:35797,127.0.71.87:35253,127.0.71.88:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:37331
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=34139
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:10.611472 29885 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:10.612855 29885 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:10.622124 29891 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:10.622306 29892 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:10.623512 29894 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:10.623953 29885 server_base.cc:1061] running on GCE node
I20260929 01:56:10.624562 29885 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:10.625732 29885 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:10.627049 29885 hybrid_clock.cc:648] HybridClock initialized: now 1790646970626913 us; error 100 us; skew 500 ppm
I20260929 01:56:10.627379 29885 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:56:10 dist-test-slave-vz23 krb5kdc[28884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:56:10.631280 29885 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:56:10.633159 29885 webserver.cc:533] Webserver started at http://127.0.71.65:34139/ using document root <none> and password file <none>
I20260929 01:56:10.633867 29885 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:10.634018 29885 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:10.639376 29885 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:56:10.642462 29901 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:10.643824 29885 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:56:10.644017 29885 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "a14c7d2a0afb4161926cd8cce4fbde4e"
format_stamp: "Formatted at 2026-09-29 01:55:49 on dist-test-slave-vz23"
I20260929 01:56:10.644816 29885 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:10.668046 29885 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:10.672305 29885 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:10.672659 29885 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:10.673719 29885 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:56:10.675227 29885 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:56:10.675355 29885 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:10.675485 29885 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:56:10.675581 29885 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:10.714198 29885 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:37331
I20260929 01:56:10.714310 30013 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:37331 every 8 connection(s)
I20260929 01:56:10.715812 29885 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:56:10.719606 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 29885
Sep 29 01:56:10 dist-test-slave-vz23 krb5kdc[28884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646955, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for hive/127.0.0.1@KRBTEST.COM
Sep 29 01:56:10 dist-test-slave-vz23 krb5kdc[28884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:56:10 dist-test-slave-vz23 krb5kdc[28884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:56:10 dist-test-slave-vz23 krb5kdc[28884](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646970, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:56:10.778633 30026 raft_consensus.cc:493] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e128f9c5227a486482933c152890ec12)
I20260929 01:56:10.778857 30026 raft_consensus.cc:515] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } }
I20260929 01:56:10.782439 30015 heartbeater.cc:344] Connected to a master server at 127.0.71.126:44289
I20260929 01:56:10.782907 30015 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:10.784080 30015 heartbeater.cc:507] Master 127.0.71.126:44289 requested a full tablet report, sending...
I20260929 01:56:10.786535 29826 ts_manager.cc:194] Registered new tserver with Master: a14c7d2a0afb4161926cd8cce4fbde4e (127.0.71.65:37331)
I20260929 01:56:10.788084 30014 heartbeater.cc:344] Connected to a master server at 127.0.71.124:36311
I20260929 01:56:10.788225 30014 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:10.788910 30014 heartbeater.cc:507] Master 127.0.71.124:36311 requested a full tablet report, sending...
I20260929 01:56:10.790221 29297 ts_manager.cc:194] Re-registered known tserver with Master: a14c7d2a0afb4161926cd8cce4fbde4e (127.0.71.65:37331)
I20260929 01:56:10.792626 30026 leader_election.cc:290] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e128f9c5227a486482933c152890ec12 (127.0.71.126:44289), 651d58de18b84e00a50539ad4f986859 (127.0.71.124:36311)
I20260929 01:56:10.792945 29641 heartbeater.cc:344] Connected to a master server at 127.0.71.126:44289
I20260929 01:56:10.793203 29641 heartbeater.cc:499] Master 127.0.71.126:44289 was elected leader, sending a full tablet report...
I20260929 01:56:10.794270 29826 master_service.cc:438] Got heartbeat from unknown tserver (permanent_uuid: "ee03cbc827f945e58402b9cfa92b92e4" instance_seqno: 1790646968667693) as {username='kudu', principal='kudu/127.0.71.66@KRBTEST.COM'} at 127.0.71.66:45533; Asking this server to re-register.
I20260929 01:56:10.795046 29641 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:10.796335 29826 ts_manager.cc:194] Registered new tserver with Master: ee03cbc827f945e58402b9cfa92b92e4 (127.0.71.66:45059)
I20260929 01:56:10.796695 30016 heartbeater.cc:344] Connected to a master server at 127.0.71.125:38709
I20260929 01:56:10.796864 30016 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:10.797435 30016 heartbeater.cc:507] Master 127.0.71.125:38709 requested a full tablet report, sending...
I20260929 01:56:10.799000 29207 ts_manager.cc:194] Re-registered known tserver with Master: a14c7d2a0afb4161926cd8cce4fbde4e (127.0.71.65:37331)
I20260929 01:56:10.804689 29836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0620fc4eb154a439efacce291731d63" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e128f9c5227a486482933c152890ec12" is_pre_election: true
I20260929 01:56:10.805326 29836 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [term 2 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d0620fc4eb154a439efacce291731d63 in term 2.
I20260929 01:56:10.806001 29192 leader_election.cc:304] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: d0620fc4eb154a439efacce291731d63, e128f9c5227a486482933c152890ec12; no voters:
I20260929 01:56:10.806504 30026 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [term 2 FOLLOWER]: Leader pre-election won for term 3
I20260929 01:56:10.806663 30026 raft_consensus.cc:493] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [term 2 FOLLOWER]: Starting leader election (detected failure of leader e128f9c5227a486482933c152890ec12)
I20260929 01:56:10.806777 30026 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:56:10.807420 29306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0620fc4eb154a439efacce291731d63" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "651d58de18b84e00a50539ad4f986859" is_pre_election: true
I20260929 01:56:10.810179 30026 raft_consensus.cc:515] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [term 3 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } }
I20260929 01:56:10.811630 29836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0620fc4eb154a439efacce291731d63" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e128f9c5227a486482933c152890ec12"
I20260929 01:56:10.811800 29306 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "d0620fc4eb154a439efacce291731d63" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "651d58de18b84e00a50539ad4f986859"
I20260929 01:56:10.811856 29836 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:56:10.811064 30026 leader_election.cc:290] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [CANDIDATE]: Term 3 election: Requested vote from peers e128f9c5227a486482933c152890ec12 (127.0.71.126:44289), 651d58de18b84e00a50539ad4f986859 (127.0.71.124:36311)
I20260929 01:56:10.817822 29836 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [term 3 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d0620fc4eb154a439efacce291731d63 in term 3.
I20260929 01:56:10.818444 29192 leader_election.cc:304] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [CANDIDATE]: Term 3 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: d0620fc4eb154a439efacce291731d63, e128f9c5227a486482933c152890ec12; no voters: 651d58de18b84e00a50539ad4f986859
I20260929 01:56:10.818892 30026 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [term 3 FOLLOWER]: Leader election won for term 3
I20260929 01:56:10.819700 30026 raft_consensus.cc:697] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [term 3 LEADER]: Becoming Leader. State: Replica: d0620fc4eb154a439efacce291731d63, State: Running, Role: LEADER
I20260929 01:56:10.820192 30026 consensus_queue.cc:237] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 2.5, Last appended by leader: 5, Current term: 3, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } }
I20260929 01:56:10.822127 30029 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0620fc4eb154a439efacce291731d63. Latest consensus state: current_term: 3 leader_uuid: "d0620fc4eb154a439efacce291731d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:56:10.822392 30029 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:10.829628 30031 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:56:10.830714 30032 raft_consensus.cc:493] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [term 2 FOLLOWER]: Starting pre-election (detected failure of leader e128f9c5227a486482933c152890ec12)
I20260929 01:56:10.831333 30032 raft_consensus.cc:515] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [term 2 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } }
I20260929 01:56:10.833627 30031 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:56:10.835417 30031 catalog_manager.cc:1308] Loaded cluster ID: 63a1f8021ce442519704c5b15912a2a3
I20260929 01:56:10.835630 30031 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:56:10.838524 30031 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:56:10.840056 30031 catalog_manager.cc:6230] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63: Loaded TSK: 0
I20260929 01:56:10.841236 29217 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "651d58de18b84e00a50539ad4f986859" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "d0620fc4eb154a439efacce291731d63" is_pre_election: true
I20260929 01:56:10.841423 30031 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:56:10.843076 30032 leader_election.cc:290] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [CANDIDATE]: Term 3 pre-election: Requested pre-vote from peers e128f9c5227a486482933c152890ec12 (127.0.71.126:44289), d0620fc4eb154a439efacce291731d63 (127.0.71.125:38709)
I20260929 01:56:10.857337 29836 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "651d58de18b84e00a50539ad4f986859" candidate_term: 3 candidate_status { last_received { term: 2 index: 5 } } ignore_live_leader: false dest_uuid: "e128f9c5227a486482933c152890ec12" is_pre_election: true
I20260929 01:56:10.857764 29836 raft_consensus.cc:2393] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [term 3 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 651d58de18b84e00a50539ad4f986859 in current term 3: Already voted for candidate d0620fc4eb154a439efacce291731d63 in this term.
I20260929 01:56:10.858804 29282 leader_election.cc:304] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [CANDIDATE]: Term 3 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 651d58de18b84e00a50539ad4f986859; no voters: d0620fc4eb154a439efacce291731d63, e128f9c5227a486482933c152890ec12
I20260929 01:56:10.859699 30032 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [term 2 FOLLOWER]: Advancing to term 3
I20260929 01:56:10.862821 30032 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [term 3 FOLLOWER]: Leader pre-election lost for term 3. Reason: could not achieve majority
I20260929 01:56:10.917105 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 29885
W20260929 01:56:10.918408 30037 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:56:10.914617 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:60260 (local address 127.0.71.65:37331)
0929 01:56:10.915391 (+ 774us) server_negotiation.cc:207] Beginning negotiation
0929 01:56:10.915399 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0929 01:56:10.915429 (+ 30us) server_negotiation.cc:408] Connection header received
0929 01:56:10.915496 (+ 67us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:56:10.915503 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:56:10.915709 (+ 206us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:56:10.915882 (+ 173us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:56:10.917096 (+ 1214us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60260: BlockingRecv error: recv got EOF from 127.0.0.1:60260 (error 108)
Metrics: {"server-negotiator.queue_time_us":622,"thread_start_us":440,"threads_started":1}
W20260929 01:56:10.919654 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 29885: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:37331: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:56:10.919801 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 29885
I20260929 01:56:10.933455 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:37331
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--webserver_port=34139
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:37059,127.0.71.85:38555,127.0.71.86:35797,127.0.71.87:35253,127.0.71.88:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:56:11.245378 29306 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [term 3 FOLLOWER]: Refusing update from remote peer d0620fc4eb154a439efacce291731d63: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 01:56:11.246232 30029 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:56:11.248670 30032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0620fc4eb154a439efacce291731d63. Latest consensus state: current_term: 3 leader_uuid: "d0620fc4eb154a439efacce291731d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:56:11.249035 30032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:11.251569 30029 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "d0620fc4eb154a439efacce291731d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:56:11.252599 30032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "d0620fc4eb154a439efacce291731d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:56:11.252785 29836 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [term 3 FOLLOWER]: Refusing update from remote peer d0620fc4eb154a439efacce291731d63: Log matching property violated. Preceding OpId in replica: term: 2 index: 5. Preceding OpId from leader: term: 3 index: 6. (index mismatch)
I20260929 01:56:11.252873 30032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 651d58de18b84e00a50539ad4f986859 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:11.251847 30029 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:11.253661 30029 consensus_queue.cc:1048] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [LEADER]: Connected to new peer: Peer: permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:56:11.261672 30046 sys_catalog.cc:455] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 3 leader_uuid: "d0620fc4eb154a439efacce291731d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:56:11.262483 30046 sys_catalog.cc:458] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:11.262656 30043 sys_catalog.cc:455] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d0620fc4eb154a439efacce291731d63. Latest consensus state: current_term: 3 leader_uuid: "d0620fc4eb154a439efacce291731d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:56:11.262897 30043 sys_catalog.cc:458] T 00000000000000000000000000000000 P e128f9c5227a486482933c152890ec12 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:56:11.314018 30039 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:11.314437 30039 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:11.314604 30039 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:56:11.314704 30039 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:11.324128 30039 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:11.324532 30039 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:56:11.336772 30039 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:37059,127.0.71.85:38555,127.0.71.86:35797,127.0.71.87:35253,127.0.71.88:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:37331
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=34139
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:11.338593 30039 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:11.340147 30039 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:11.351153 30052 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:11.351204 30051 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:11.352208 30054 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:11.352749 30039 server_base.cc:1061] running on GCE node
I20260929 01:56:11.353425 30039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:11.354635 30039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:11.356058 30039 hybrid_clock.cc:648] HybridClock initialized: now 1790646971355869 us; error 108 us; skew 500 ppm
I20260929 01:56:11.356606 30039 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:56:11 dist-test-slave-vz23 krb5kdc[28884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:56:11.361129 30039 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:56:11.362947 30039 webserver.cc:533] Webserver started at http://127.0.71.65:34139/ using document root <none> and password file <none>
I20260929 01:56:11.363763 30039 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:11.363895 30039 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:11.370352 30039 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:56:11.374269 30061 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:11.376325 30039 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:56:11.376603 30039 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "a14c7d2a0afb4161926cd8cce4fbde4e"
format_stamp: "Formatted at 2026-09-29 01:55:49 on dist-test-slave-vz23"
I20260929 01:56:11.377401 30039 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:11.392561 30039 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:11.407758 30039 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:11.408113 30039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:11.409307 30039 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:56:11.411096 30039 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:56:11.411176 30039 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:11.411314 30039 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:56:11.411399 30039 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:56:11.411891 30039 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:56:11.411980 30039 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:56:11.412089 30039 fault_injection.cc:59] injecting fault for kudu (pid 30039): FLAGS_fault_before_start (process will exit)
I20260929 01:56:11.431699 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:37331
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--webserver_port=34139
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:37059,127.0.71.85:38555,127.0.71.86:35797,127.0.71.87:35253,127.0.71.88:34951
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:56:11.663548 30029 sys_catalog.cc:455] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 3 leader_uuid: "d0620fc4eb154a439efacce291731d63" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e128f9c5227a486482933c152890ec12" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44289 } } peers { permanent_uuid: "d0620fc4eb154a439efacce291731d63" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38709 } } peers { permanent_uuid: "651d58de18b84e00a50539ad4f986859" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36311 } } }
I20260929 01:56:11.663892 30029 sys_catalog.cc:458] T 00000000000000000000000000000000 P d0620fc4eb154a439efacce291731d63 [sys.catalog]: This master's current role is: LEADER
W20260929 01:56:11.798987 30069 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:11.799439 30069 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:11.799548 30069 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:11.809147 30069 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:11.809387 30069 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:56:11.821352 30069 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:37059,127.0.71.85:38555,127.0.71.86:35797,127.0.71.87:35253,127.0.71.88:34951
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:37331
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=34139
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:44289,127.0.71.125:38709,127.0.71.124:36311
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:11.822966 30069 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:11.824321 30069 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:11.835590 30076 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:11.835860 30078 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:11.835927 30075 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:11.836848 30069 server_base.cc:1061] running on GCE node
I20260929 01:56:11.837509 30069 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:11.838759 30069 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:11.840092 30069 hybrid_clock.cc:648] HybridClock initialized: now 1790646971839933 us; error 123 us; skew 500 ppm
I20260929 01:56:11.840505 30069 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:56:11 dist-test-slave-vz23 krb5kdc[28884](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790646971, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:56:11.844599 30069 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:56:11.846390 30069 webserver.cc:533] Webserver started at http://127.0.71.65:34139/ using document root <none> and password file <none>
I20260929 01:56:11.847110 30069 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:11.847216 30069 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:11.853581 30069 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:56:11.856773 30085 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:11.858677 30069 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.001s
I20260929 01:56:11.858851 30069 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "a14c7d2a0afb4161926cd8cce4fbde4e"
format_stamp: "Formatted at 2026-09-29 01:55:49 on dist-test-slave-vz23"
I20260929 01:56:11.859616 30069 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:11.877909 30069 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:11.882318 30069 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:11.882632 30069 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:11.883731 30069 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:56:11.885838 30069 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:56:11.885936 30069 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:11.886112 30069 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:56:11.886196 30069 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:11.924141 30069 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:37331
I20260929 01:56:11.924235 30197 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:37331 every 8 connection(s)
I20260929 01:56:11.925745 30069 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestBasicOperation_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:56:11.930408 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 30069
I20260929 01:56:11.930814 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 30069
W20260929 01:56:11.949925 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 30069: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:37331: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:56:11.950138 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 30069
I20260929 01:56:11.967463 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 29512
W20260929 01:56:11.967546 30203 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:56:11.929707 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:35845 (local address 127.0.71.126:44289)
0929 01:56:11.951448 (+ 21741us) server_negotiation.cc:207] Beginning negotiation
0929 01:56:11.951457 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0929 01:56:11.951482 (+ 25us) server_negotiation.cc:408] Connection header received
0929 01:56:11.951554 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:56:11.951559 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:56:11.951646 (+ 87us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:56:11.951763 (+ 117us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:56:11.967161 (+ 15398us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:35845: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":21580,"thread_start_us":21784,"threads_started":1}
W20260929 01:56:11.968228 30209 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:56:11.942260 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:39761 (local address 127.0.71.125:38709)
0929 01:56:11.945969 (+ 3709us) server_negotiation.cc:207] Beginning negotiation
0929 01:56:11.945976 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0929 01:56:11.967161 (+ 21185us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:39761: BlockingRecv error: recv got EOF from 127.0.71.65:39761 (error 108)
Metrics: {"server-negotiator.queue_time_us":3592,"thread_start_us":3480,"threads_started":1}
W20260929 01:56:11.968261 30202 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:56:11.929307 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:55277 (local address 127.0.71.124:36311)
0929 01:56:11.946392 (+ 17085us) server_negotiation.cc:207] Beginning negotiation
0929 01:56:11.946401 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0929 01:56:11.946427 (+ 26us) server_negotiation.cc:408] Connection header received
0929 01:56:11.946502 (+ 75us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:56:11.946508 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:56:11.946590 (+ 82us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:56:11.946724 (+ 134us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:56:11.967250 (+ 20526us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:55277: BlockingRecv error: recv got EOF from 127.0.71.65:55277 (error 108)
Metrics: {"server-negotiator.queue_time_us":16928,"thread_start_us":16757,"threads_started":1}
W20260929 01:56:11.970469 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 29512: Not authorized: Client connection negotiation failed: client connection to 127.0.71.66:45059: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:56:11.970631 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 29512
I20260929 01:56:11.988041 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 29654
W20260929 01:56:11.991343 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 29654: Not authorized: Client connection negotiation failed: client connection to 127.0.71.67:43509: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:56:11.991554 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 29654
I20260929 01:56:12.009442 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 29793
W20260929 01:56:12.012008 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 29793: Not authorized: Client connection negotiation failed: client connection to 127.0.71.126:44289: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:56:12.012167 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 29793
W20260929 01:56:12.012185 30203 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:56:12.010337 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:34956 (local address 127.0.71.126:44289)
0929 01:56:12.010524 (+ 187us) server_negotiation.cc:207] Beginning negotiation
0929 01:56:12.010529 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0929 01:56:12.010567 (+ 38us) server_negotiation.cc:408] Connection header received
0929 01:56:12.010820 (+ 253us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:56:12.010825 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:56:12.010920 (+ 95us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:56:12.011028 (+ 108us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:56:12.011950 (+ 922us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34956: BlockingRecv error: recv got EOF from 127.0.0.1:34956 (error 108)
Metrics: {"server-negotiator.queue_time_us":72}
I20260929 01:56:12.032580 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 29175
I20260929 01:56:12.033749 29237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:12.174047 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 29175
I20260929 01:56:12.195866 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 29265
I20260929 01:56:12.197268 29327 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:12.311295 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 29265
Shutting down hive metastore.
2026-09-29T01:56:12Z chronyd exiting
2026-09-29T01:56:12Z chronyd exiting
2026-09-29T01:56:12Z chronyd exiting
2026-09-29T01:56:12Z chronyd exiting
2026-09-29T01:56:12Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestBasicOperation/11 (47418 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/0
I20260929 01:56:12.785665 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:35327
--webserver_interface=127.0.71.126
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:35327,127.0.71.125:35679,127.0.71.124:39001 with env {}
W20260929 01:56:13.163041 30219 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:13.163552 30219 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:13.163781 30219 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:13.174139 30219 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:56:13.174273 30219 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:13.174346 30219 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:56:13.174392 30219 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:56:13.185508 30219 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:35327,127.0.71.125:35679,127.0.71.124:39001
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:35327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:13.187274 30219 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:13.189404 30219 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:13.200575 30225 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:13.200726 30227 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:13.201318 30219 server_base.cc:1061] running on GCE node
W20260929 01:56:13.201637 30224 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:13.202630 30219 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:56:13.202808 30219 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:56:13.202883 30219 hybrid_clock.cc:648] HybridClock initialized: now 1790646973202863 us; error 0 us; skew 500 ppm
I20260929 01:56:13.203222 30219 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:13.206948 30219 webserver.cc:533] Webserver started at http://127.0.71.126:34393/ using document root <none> and password file <none>
I20260929 01:56:13.207737 30219 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:13.207854 30219 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:13.208211 30219 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:56:13.210987 30219 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "7907493231354841a2732fbae6a131b4"
format_stamp: "Formatted at 2026-09-29 01:56:13 on dist-test-slave-vz23"
I20260929 01:56:13.211900 30219 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "7907493231354841a2732fbae6a131b4"
format_stamp: "Formatted at 2026-09-29 01:56:13 on dist-test-slave-vz23"
I20260929 01:56:13.218513 30219 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260929 01:56:13.223497 30232 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:13.225953 30219 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260929 01:56:13.226202 30219 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "7907493231354841a2732fbae6a131b4"
format_stamp: "Formatted at 2026-09-29 01:56:13 on dist-test-slave-vz23"
I20260929 01:56:13.226459 30219 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:13.250725 30219 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:13.251765 30219 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:13.252115 30219 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:13.276676 30219 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:35327
I20260929 01:56:13.276715 30283 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:35327 every 8 connection(s)
I20260929 01:56:13.278309 30219 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:56:13.279752 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 30219
I20260929 01:56:13.280015 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:56:13.281666 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:35679
--webserver_interface=127.0.71.125
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:35327,127.0.71.125:35679,127.0.71.124:39001 with env {}
I20260929 01:56:13.284482 30284 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:56:13.293511 30284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:13.311870 30284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:56:13.313789 30234 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.125:35679: connect: Connection refused (error 111)
W20260929 01:56:13.316005 30284 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:35679: Network error: Client connection negotiation failed: client connection to 127.0.71.125:35679: connect: Connection refused (error 111)
I20260929 01:56:13.365463 30284 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } attempt: 1
W20260929 01:56:13.368709 30284 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:35679: Network error: Client connection negotiation failed: client connection to 127.0.71.125:35679: connect: Connection refused (error 111)
I20260929 01:56:13.437151 30284 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } attempt: 2
W20260929 01:56:13.439631 30284 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:35679: Network error: Client connection negotiation failed: client connection to 127.0.71.125:35679: connect: Connection refused (error 111)
I20260929 01:56:13.530975 30284 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } attempt: 3
W20260929 01:56:13.533187 30284 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:35679: Network error: Client connection negotiation failed: client connection to 127.0.71.125:35679: connect: Connection refused (error 111)
W20260929 01:56:13.655402 30286 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:13.655776 30286 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:13.655854 30286 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:13.664516 30286 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:56:13.664636 30286 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:13.664697 30286 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:56:13.664749 30286 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:56:13.675488 30286 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:35327,127.0.71.125:35679,127.0.71.124:39001
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:35679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:13.676523 30284 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } attempt: 4
I20260929 01:56:13.677593 30286 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:13.679716 30286 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:13.679934 30284 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:35679: Network error: Client connection negotiation failed: client connection to 127.0.71.125:35679: connect: Connection refused (error 111)
W20260929 01:56:13.690456 30294 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:13.690403 30297 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:13.690683 30295 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:13.691532 30286 server_base.cc:1061] running on GCE node
I20260929 01:56:13.692505 30286 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:56:13.692740 30286 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:56:13.692853 30286 hybrid_clock.cc:648] HybridClock initialized: now 1790646973692808 us; error 0 us; skew 500 ppm
I20260929 01:56:13.693265 30286 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:13.696835 30286 webserver.cc:533] Webserver started at http://127.0.71.125:46063/ using document root <none> and password file <none>
I20260929 01:56:13.697794 30286 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:13.697916 30286 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:13.698326 30286 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:56:13.701149 30286 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-1/data/instance:
uuid: "f8ed66573e594723a8f14f4794e3a138"
format_stamp: "Formatted at 2026-09-29 01:56:13 on dist-test-slave-vz23"
I20260929 01:56:13.702147 30286 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-1/wal/instance:
uuid: "f8ed66573e594723a8f14f4794e3a138"
format_stamp: "Formatted at 2026-09-29 01:56:13 on dist-test-slave-vz23"
I20260929 01:56:13.709084 30286 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.005s
I20260929 01:56:13.714396 30302 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:13.716691 30286 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:56:13.716964 30286 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "f8ed66573e594723a8f14f4794e3a138"
format_stamp: "Formatted at 2026-09-29 01:56:13 on dist-test-slave-vz23"
I20260929 01:56:13.717276 30286 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:13.735514 30286 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:13.736799 30286 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:13.737164 30286 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:13.763681 30286 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:35679
I20260929 01:56:13.763708 30353 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:35679 every 8 connection(s)
I20260929 01:56:13.765679 30286 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:56:13.770941 30354 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:56:13.775233 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 30286
I20260929 01:56:13.775537 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:56:13.776952 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:39001
--webserver_interface=127.0.71.124
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:35327,127.0.71.125:35679,127.0.71.124:39001 with env {}
I20260929 01:56:13.778644 30354 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:13.796847 30354 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:13.811559 30354 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:56:13.813027 30304 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.124:39001: connect: Connection refused (error 111)
W20260929 01:56:13.815387 30354 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:39001: Network error: Client connection negotiation failed: client connection to 127.0.71.124:39001: connect: Connection refused (error 111)
I20260929 01:56:13.864895 30354 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } attempt: 1
W20260929 01:56:13.867693 30354 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:39001: Network error: Client connection negotiation failed: client connection to 127.0.71.124:39001: connect: Connection refused (error 111)
I20260929 01:56:13.936017 30354 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } attempt: 2
W20260929 01:56:13.939453 30354 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:39001: Network error: Client connection negotiation failed: client connection to 127.0.71.124:39001: connect: Connection refused (error 111)
I20260929 01:56:13.979286 30284 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } attempt: 5
I20260929 01:56:13.987258 30284 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:56:13.990128 30284 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:39001: Network error: Client connection negotiation failed: client connection to 127.0.71.124:39001: connect: Connection refused (error 111)
I20260929 01:56:14.030781 30354 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } attempt: 3
W20260929 01:56:14.034047 30354 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:39001: Network error: Client connection negotiation failed: client connection to 127.0.71.124:39001: connect: Connection refused (error 111)
I20260929 01:56:14.041549 30284 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } attempt: 1
W20260929 01:56:14.044382 30284 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:39001: Network error: Client connection negotiation failed: client connection to 127.0.71.124:39001: connect: Connection refused (error 111)
I20260929 01:56:14.112804 30284 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } attempt: 2
W20260929 01:56:14.115131 30284 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:39001: Network error: Client connection negotiation failed: client connection to 127.0.71.124:39001: connect: Connection refused (error 111)
I20260929 01:56:14.177419 30354 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } attempt: 4
W20260929 01:56:14.180174 30357 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:14.180756 30357 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:14.180946 30357 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:14.180923 30354 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:39001: Network error: Client connection negotiation failed: client connection to 127.0.71.124:39001: connect: Connection refused (error 111)
W20260929 01:56:14.190435 30357 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:56:14.190606 30357 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:14.190738 30357 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:56:14.190832 30357 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:56:14.202760 30357 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:35327,127.0.71.125:35679,127.0.71.124:39001
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:39001
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:14.204726 30357 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:14.206787 30357 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:14.219607 30365 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:14.219959 30367 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:14.219993 30364 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:14.220708 30357 server_base.cc:1061] running on GCE node
I20260929 01:56:14.221413 30284 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } attempt: 3
I20260929 01:56:14.221815 30357 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:56:14.222023 30357 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:56:14.222146 30357 hybrid_clock.cc:648] HybridClock initialized: now 1790646974222104 us; error 0 us; skew 500 ppm
I20260929 01:56:14.222558 30357 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
W20260929 01:56:14.223951 30284 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:39001: Network error: Client connection negotiation failed: client connection to 127.0.71.124:39001: connect: Connection refused (error 111)
I20260929 01:56:14.227006 30357 webserver.cc:533] Webserver started at http://127.0.71.124:34915/ using document root <none> and password file <none>
I20260929 01:56:14.227967 30357 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:14.228081 30357 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:14.228636 30357 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:56:14.234285 30357 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-2/data/instance:
uuid: "bf63a07c4759445a9c28ef8a8326165d"
format_stamp: "Formatted at 2026-09-29 01:56:14 on dist-test-slave-vz23"
I20260929 01:56:14.235302 30357 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-2/wal/instance:
uuid: "bf63a07c4759445a9c28ef8a8326165d"
format_stamp: "Formatted at 2026-09-29 01:56:14 on dist-test-slave-vz23"
I20260929 01:56:14.242748 30357 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260929 01:56:14.247184 30372 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:14.249451 30357 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 01:56:14.249745 30357 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "bf63a07c4759445a9c28ef8a8326165d"
format_stamp: "Formatted at 2026-09-29 01:56:14 on dist-test-slave-vz23"
I20260929 01:56:14.250341 30357 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:14.284490 30357 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:14.285776 30357 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:14.286135 30357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:14.318554 30357 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:39001
I20260929 01:56:14.318580 30423 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:39001 every 8 connection(s)
I20260929 01:56:14.320398 30357 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:56:14.324760 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 30357
I20260929 01:56:14.325448 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:56:14.326347 30424 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:56:14.335873 30424 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:14.352111 30424 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:14.361155 30424 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:14.375887 30424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d: Bootstrap starting.
I20260929 01:56:14.381390 30424 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d: Neither blocks nor log segments found. Creating new log.
I20260929 01:56:14.383314 30424 log.cc:826] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:56:14.388073 30424 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d: No bootstrap required, opened a new log
I20260929 01:56:14.395680 30424 raft_consensus.cc:359] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } }
I20260929 01:56:14.396147 30424 raft_consensus.cc:385] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:56:14.396306 30424 raft_consensus.cc:740] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: bf63a07c4759445a9c28ef8a8326165d, State: Initialized, Role: FOLLOWER
I20260929 01:56:14.397198 30424 consensus_queue.cc:260] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } }
I20260929 01:56:14.398470 30431 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } }
I20260929 01:56:14.398876 30431 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:14.399775 30424 sys_catalog.cc:565] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:56:14.401235 30284 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } attempt: 4
I20260929 01:56:14.411746 30357 master_runner.cc:187] Error getting master registration for 127.0.71.126:35327: OK, instance_id { permanent_uuid: "7907493231354841a2732fbae6a131b4" instance_seqno: 1790646973270246 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:56:14.421033 30357 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:56:14.423935 30284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4: Bootstrap starting.
W20260929 01:56:14.425514 30442 catalog_manager.cc:1607] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:56:14.425873 30442 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:56:14.429776 30284 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4: Neither blocks nor log segments found. Creating new log.
I20260929 01:56:14.431711 30284 log.cc:826] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4: Log is configured to *not* fsync() on all Append() calls
I20260929 01:56:14.436048 30284 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4: No bootstrap required, opened a new log
I20260929 01:56:14.445853 30284 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } }
I20260929 01:56:14.446389 30284 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:56:14.446553 30284 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7907493231354841a2732fbae6a131b4, State: Initialized, Role: FOLLOWER
I20260929 01:56:14.447533 30284 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } }
I20260929 01:56:14.449560 30445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } }
I20260929 01:56:14.449993 30445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:14.451016 30284 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:56:14.469686 30456 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:56:14.470031 30456 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:56:14.468181 30219 master_runner.cc:187] Error getting master registration for 127.0.71.125:35679: OK, instance_id { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" instance_seqno: 1790646973757247 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:56:14.477020 30219 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:56:14.480224 30354 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } attempt: 5
I20260929 01:56:14.498191 30354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138: Bootstrap starting.
I20260929 01:56:14.503592 30354 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138: Neither blocks nor log segments found. Creating new log.
I20260929 01:56:14.505376 30354 log.cc:826] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138: Log is configured to *not* fsync() on all Append() calls
W20260929 01:56:14.505769 30238 tablet.cc:2433] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:56:14.510689 30354 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138: No bootstrap required, opened a new log
I20260929 01:56:14.520836 30354 raft_consensus.cc:359] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } }
I20260929 01:56:14.521296 30354 raft_consensus.cc:385] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:56:14.521524 30354 raft_consensus.cc:740] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8ed66573e594723a8f14f4794e3a138, State: Initialized, Role: FOLLOWER
I20260929 01:56:14.522503 30354 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } }
I20260929 01:56:14.523895 30459 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } }
I20260929 01:56:14.524417 30459 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:14.525609 30354 sys_catalog.cc:565] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:56:14.539567 30378 tablet.cc:2433] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:56:14.539783 30470 catalog_manager.cc:818] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 01:56:14.554260 30286 master_runner.cc:264] No leader master found from master f8ed66573e594723a8f14f4794e3a138
I20260929 01:56:14.554440 30286 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:56:14.717013 30459 raft_consensus.cc:493] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:56:14.717365 30459 raft_consensus.cc:515] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } }
I20260929 01:56:14.719398 30459 leader_election.cc:290] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 7907493231354841a2732fbae6a131b4 (127.0.71.126:35327), bf63a07c4759445a9c28ef8a8326165d (127.0.71.124:39001)
I20260929 01:56:14.720965 30259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f8ed66573e594723a8f14f4794e3a138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7907493231354841a2732fbae6a131b4" is_pre_election: true
I20260929 01:56:14.721192 30399 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f8ed66573e594723a8f14f4794e3a138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf63a07c4759445a9c28ef8a8326165d" is_pre_election: true
I20260929 01:56:14.721598 30259 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8ed66573e594723a8f14f4794e3a138 in term 0.
I20260929 01:56:14.721688 30399 raft_consensus.cc:2468] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8ed66573e594723a8f14f4794e3a138 in term 0.
I20260929 01:56:14.722424 30304 leader_election.cc:304] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7907493231354841a2732fbae6a131b4, f8ed66573e594723a8f14f4794e3a138; no voters:
I20260929 01:56:14.723461 30459 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:56:14.723642 30459 raft_consensus.cc:493] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:56:14.723742 30459 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:56:14.727672 30459 raft_consensus.cc:515] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } }
I20260929 01:56:14.728742 30459 leader_election.cc:290] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [CANDIDATE]: Term 1 election: Requested vote from peers 7907493231354841a2732fbae6a131b4 (127.0.71.126:35327), bf63a07c4759445a9c28ef8a8326165d (127.0.71.124:39001)
I20260929 01:56:14.729405 30259 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f8ed66573e594723a8f14f4794e3a138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7907493231354841a2732fbae6a131b4"
I20260929 01:56:14.729547 30399 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f8ed66573e594723a8f14f4794e3a138" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "bf63a07c4759445a9c28ef8a8326165d"
I20260929 01:56:14.729687 30259 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:56:14.729791 30399 raft_consensus.cc:3060] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:56:14.733844 30259 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8ed66573e594723a8f14f4794e3a138 in term 1.
I20260929 01:56:14.734120 30399 raft_consensus.cc:2468] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8ed66573e594723a8f14f4794e3a138 in term 1.
I20260929 01:56:14.734597 30304 leader_election.cc:304] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7907493231354841a2732fbae6a131b4, f8ed66573e594723a8f14f4794e3a138; no voters:
I20260929 01:56:14.735038 30459 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:56:14.735642 30459 raft_consensus.cc:697] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [term 1 LEADER]: Becoming Leader. State: Replica: f8ed66573e594723a8f14f4794e3a138, State: Running, Role: LEADER
I20260929 01:56:14.736285 30459 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } }
W20260929 01:56:14.740976 30308 tablet.cc:2433] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:56:14.747432 30471 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8ed66573e594723a8f14f4794e3a138. Latest consensus state: current_term: 1 leader_uuid: "f8ed66573e594723a8f14f4794e3a138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } }
I20260929 01:56:14.747989 30471 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:14.749289 30473 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:56:14.757110 30473 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:56:14.782121 30399 raft_consensus.cc:1275] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d [term 1 FOLLOWER]: Refusing update from remote peer f8ed66573e594723a8f14f4794e3a138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:56:14.782161 30259 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [term 1 FOLLOWER]: Refusing update from remote peer f8ed66573e594723a8f14f4794e3a138: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:56:14.784106 30471 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:56:14.785686 30471 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:56:14.819612 30431 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8ed66573e594723a8f14f4794e3a138. Latest consensus state: current_term: 1 leader_uuid: "f8ed66573e594723a8f14f4794e3a138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } }
I20260929 01:56:14.819976 30431 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:14.824580 30475 mvcc.cc:204] Tried to move back new op lower bound from 7334490008695971840 to 7334490008527720448. Current Snapshot: MvccSnapshot[applied={T|T < 7334490008695971840}]
I20260929 01:56:14.826610 30445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8ed66573e594723a8f14f4794e3a138. Latest consensus state: current_term: 1 leader_uuid: "f8ed66573e594723a8f14f4794e3a138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } }
I20260929 01:56:14.826965 30445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:14.827248 30476 mvcc.cc:204] Tried to move back new op lower bound from 7334490008695971840 to 7334490008527720448. Current Snapshot: MvccSnapshot[applied={T|T < 7334490008695971840}]
I20260929 01:56:14.837131 30477 mvcc.cc:204] Tried to move back new op lower bound from 7334490008695971840 to 7334490008527720448. Current Snapshot: MvccSnapshot[applied={T|T < 7334490008695971840}]
I20260929 01:56:14.842314 30431 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8ed66573e594723a8f14f4794e3a138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } }
I20260929 01:56:14.842612 30431 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:14.842741 30459 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8ed66573e594723a8f14f4794e3a138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } }
I20260929 01:56:14.843076 30459 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:14.843474 30459 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8ed66573e594723a8f14f4794e3a138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } }
I20260929 01:56:14.843731 30459 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:14.853880 30473 catalog_manager.cc:1396] Generated new cluster ID: 28270ee6cb5d4aec85e53ccce360352e
I20260929 01:56:14.854048 30473 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:56:14.854514 30445 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8ed66573e594723a8f14f4794e3a138" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } }
I20260929 01:56:14.854776 30445 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:14.881202 30473 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 01:56:14.883589 30473 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:56:14.914706 30473 catalog_manager.cc:6219] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138: Generated new TSK 0
I20260929 01:56:14.916277 30473 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:56:14.930805 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:35327,127.0.71.125:35679,127.0.71.124:39001
--time_source=system_unsync with env {}
I20260929 01:56:15.429275 30442 catalog_manager.cc:1308] Loaded cluster ID: 28270ee6cb5d4aec85e53ccce360352e
I20260929 01:56:15.429632 30442 catalog_manager.cc:1601] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d: loading cluster ID for follower catalog manager: success
I20260929 01:56:15.433816 30442 catalog_manager.cc:1623] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d: acquiring CA information for follower catalog manager: success
I20260929 01:56:15.436848 30442 catalog_manager.cc:1651] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 01:56:15.445808 30484 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:15.446281 30484 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:15.446431 30484 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:15.456527 30484 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:15.456825 30484 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:56:15.468705 30484 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:35327,127.0.71.125:35679,127.0.71.124:39001
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:15.470678 30484 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:15.473384 30484 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260929 01:56:15.473717 30456 catalog_manager.cc:1308] Loaded cluster ID: 28270ee6cb5d4aec85e53ccce360352e
I20260929 01:56:15.473843 30456 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4: loading cluster ID for follower catalog manager: success
I20260929 01:56:15.478118 30456 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4: acquiring CA information for follower catalog manager: success
I20260929 01:56:15.480861 30456 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 01:56:15.489058 30490 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:15.489071 30489 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:15.489238 30492 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:15.491308 30484 server_base.cc:1061] running on GCE node
I20260929 01:56:15.492908 30484 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:56:15.493242 30484 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:56:15.493386 30484 hybrid_clock.cc:648] HybridClock initialized: now 1790646975493355 us; error 0 us; skew 500 ppm
I20260929 01:56:15.494115 30484 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:15.497598 30484 webserver.cc:533] Webserver started at http://127.0.71.65:42151/ using document root <none> and password file <none>
I20260929 01:56:15.498586 30484 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:15.498783 30484 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:15.499250 30484 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:56:15.502560 30484 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "abb492f02115498487fc53f1ee17886f"
format_stamp: "Formatted at 2026-09-29 01:56:15 on dist-test-slave-vz23"
I20260929 01:56:15.503664 30484 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "abb492f02115498487fc53f1ee17886f"
format_stamp: "Formatted at 2026-09-29 01:56:15 on dist-test-slave-vz23"
I20260929 01:56:15.512204 30484 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.009s sys 0.001s
I20260929 01:56:15.517467 30497 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:15.519680 30484 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260929 01:56:15.519992 30484 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "abb492f02115498487fc53f1ee17886f"
format_stamp: "Formatted at 2026-09-29 01:56:15 on dist-test-slave-vz23"
I20260929 01:56:15.520392 30484 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:15.551697 30484 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:15.552843 30484 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:15.553139 30484 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:15.554327 30484 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:56:15.556907 30484 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:56:15.557062 30484 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:15.557219 30484 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:56:15.557331 30484 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:15.612900 30484 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:42695
I20260929 01:56:15.612900 30609 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:42695 every 8 connection(s)
I20260929 01:56:15.614709 30484 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:56:15.615983 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 30484
I20260929 01:56:15.616288 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
I20260929 01:56:15.642244 30612 heartbeater.cc:344] Connected to a master server at 127.0.71.125:35679
I20260929 01:56:15.642409 30610 heartbeater.cc:344] Connected to a master server at 127.0.71.124:39001
I20260929 01:56:15.643086 30610 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:15.643086 30612 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:15.644698 30612 heartbeater.cc:507] Master 127.0.71.125:35679 requested a full tablet report, sending...
I20260929 01:56:15.645293 30610 heartbeater.cc:507] Master 127.0.71.124:39001 requested a full tablet report, sending...
I20260929 01:56:15.649580 30611 heartbeater.cc:344] Connected to a master server at 127.0.71.126:35327
I20260929 01:56:15.649749 30319 ts_manager.cc:194] Registered new tserver with Master: abb492f02115498487fc53f1ee17886f (127.0.71.65:42695)
I20260929 01:56:15.649842 30611 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:15.649870 30389 ts_manager.cc:194] Registered new tserver with Master: abb492f02115498487fc53f1ee17886f (127.0.71.65:42695)
I20260929 01:56:15.650966 30611 heartbeater.cc:507] Master 127.0.71.126:35327 requested a full tablet report, sending...
I20260929 01:56:15.653188 30319 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:45201
I20260929 01:56:15.653939 30249 ts_manager.cc:194] Registered new tserver with Master: abb492f02115498487fc53f1ee17886f (127.0.71.65:42695)
I20260929 01:56:15.663184 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:56:15.666706 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.123:34857
--webserver_interface=127.0.71.123
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:35327,127.0.71.125:35679,127.0.71.124:39001,127.0.71.123:34857 with env {}
W20260929 01:56:16.116008 30620 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:16.116559 30620 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:16.116719 30620 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:16.127192 30620 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:56:16.127342 30620 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:16.127460 30620 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:56:16.127545 30620 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:56:16.138895 30620 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:35327,127.0.71.125:35679,127.0.71.124:39001,127.0.71.123:34857
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.123:34857
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.0.71.123
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:16.140894 30620 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:16.143046 30620 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:16.154235 30625 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:16.154238 30626 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:16.156904 30628 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:16.157812 30620 server_base.cc:1061] running on GCE node
I20260929 01:56:16.159010 30620 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:56:16.159243 30620 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:56:16.159385 30620 hybrid_clock.cc:648] HybridClock initialized: now 1790646976159350 us; error 0 us; skew 500 ppm
I20260929 01:56:16.159780 30620 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:16.163489 30620 webserver.cc:533] Webserver started at http://127.0.71.123:44525/ using document root <none> and password file <none>
I20260929 01:56:16.164412 30620 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:16.164649 30620 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:16.165115 30620 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:56:16.167884 30620 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-3/data/instance:
uuid: "6573536be5c84204886a2fdbe4a41f85"
format_stamp: "Formatted at 2026-09-29 01:56:16 on dist-test-slave-vz23"
I20260929 01:56:16.168964 30620 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-3/wal/instance:
uuid: "6573536be5c84204886a2fdbe4a41f85"
format_stamp: "Formatted at 2026-09-29 01:56:16 on dist-test-slave-vz23"
I20260929 01:56:16.175596 30620 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.004s
I20260929 01:56:16.179637 30633 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:16.181826 30620 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 01:56:16.182061 30620 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "6573536be5c84204886a2fdbe4a41f85"
format_stamp: "Formatted at 2026-09-29 01:56:16 on dist-test-slave-vz23"
I20260929 01:56:16.182322 30620 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:16.212260 30620 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:16.213665 30620 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:16.214047 30620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:16.247633 30620 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.123:34857
I20260929 01:56:16.247614 30684 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.123:34857 every 8 connection(s)
I20260929 01:56:16.250029 30620 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
I20260929 01:56:16.253871 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 30620
I20260929 01:56:16.257541 30685 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:56:16.268689 30685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:16.291875 30685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:16.303661 30685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:16.315001 30685 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.123" port: 34857 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:16.339100 30685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85: Bootstrap starting.
I20260929 01:56:16.346103 30685 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85: Neither blocks nor log segments found. Creating new log.
I20260929 01:56:16.348212 30685 log.cc:826] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85: Log is configured to *not* fsync() on all Append() calls
I20260929 01:56:16.353370 30685 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85: No bootstrap required, opened a new log
I20260929 01:56:16.365622 30685 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 34857 } }
I20260929 01:56:16.366237 30685 raft_consensus.cc:385] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:56:16.366395 30685 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 6573536be5c84204886a2fdbe4a41f85, State: Initialized, Role: FOLLOWER
I20260929 01:56:16.367568 30685 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 34857 } }
I20260929 01:56:16.368932 30694 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 34857 } } }
I20260929 01:56:16.369321 30694 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:16.371307 30685 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:56:16.385650 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 30484
W20260929 01:56:16.390174 30705 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:56:16.390386 30705 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:56:16.402747 30620 master_runner.cc:429] Detected that this master 6573536be5c84204886a2fdbe4a41f85 is joining an existing cluster
I20260929 01:56:16.402937 30620 catalog_manager.cc:1037] Shutting down the Catalog Manager
I20260929 01:56:16.404193 30620 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260929 01:56:16.404780 30620 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260929 01:56:16.405118 30620 tablet_replica.cc:333] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85: stopping tablet replica
I20260929 01:56:16.407691 30620 master_runner.cc:434] Initiating AddMaster RPC to add 127.0.71.123:34857
I20260929 01:56:16.420637 30605 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:16.485666 30319 catalog_manager.cc:7351] Initiating ChangeConfig request to add master 127.0.71.123:34857: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "6573536be5c84204886a2fdbe4a41f85"
member_type: NON_VOTER
last_known_addr {
host: "127.0.71.123"
port: 34857
}
attrs {
promote: true
}
}
dest_uuid: "f8ed66573e594723a8f14f4794e3a138"
cas_config_opid_index: -1
I20260929 01:56:16.487977 30319 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: true } }
I20260929 01:56:16.500284 30259 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [term 1 FOLLOWER]: Refusing update from remote peer f8ed66573e594723a8f14f4794e3a138: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 01:56:16.501272 30471 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260929 01:56:16.510651 30306 consensus_peers.cc:597] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 -> Peer 6573536be5c84204886a2fdbe4a41f85 (127.0.71.123:34857): Couldn't send request to peer 6573536be5c84204886a2fdbe4a41f85. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260929 01:56:16.512852 30398 raft_consensus.cc:1275] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d [term 1 FOLLOWER]: Refusing update from remote peer f8ed66573e594723a8f14f4794e3a138: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 01:56:16.513732 30471 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:56:16.540972 30720 raft_consensus.cc:2955] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6573536be5c84204886a2fdbe4a41f85 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: true } } }
I20260929 01:56:16.546141 30398 raft_consensus.cc:2955] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6573536be5c84204886a2fdbe4a41f85 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: true } } }
I20260929 01:56:16.547698 30259 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 6573536be5c84204886a2fdbe4a41f85 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: true } } }
I20260929 01:56:16.562386 30720 catalog_manager.cc:7355] Successfully completed master ChangeConfig request to add master 127.0.71.123:34857
I20260929 01:56:16.564090 30620 master.cc:562] Master@127.0.71.123:34857 shutting down...
I20260929 01:56:16.565797 30471 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f8ed66573e594723a8f14f4794e3a138" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: true } } }
I20260929 01:56:16.566175 30471 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:16.576205 30721 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8ed66573e594723a8f14f4794e3a138" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: true } } }
I20260929 01:56:16.576737 30721 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:16.577337 30721 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8ed66573e594723a8f14f4794e3a138" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: true } } }
I20260929 01:56:16.577584 30721 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:16.578661 30722 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8ed66573e594723a8f14f4794e3a138" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: true } } }
I20260929 01:56:16.579519 30722 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:16.580050 30723 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8ed66573e594723a8f14f4794e3a138" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: true } } }
I20260929 01:56:16.580502 30723 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:16.590673 30620 master.cc:584] Master@127.0.71.123:34857 shutdown complete.
I20260929 01:56:16.590907 30620 master_runner.cc:306] Clearing existing system tablet
I20260929 01:56:16.591429 30620 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:16.591610 30620 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:16.601193 30620 fs_manager.cc:714] Time spent opening directory manager: real 0.008s user 0.002s sys 0.006s
I20260929 01:56:16.616645 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 30484
I20260929 01:56:16.616874 30726 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:16.620659 30620 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:16.623405 30620 fs_manager.cc:730] Time spent opening block manager: real 0.021s user 0.005s sys 0.008s
I20260929 01:56:16.623638 30620 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "6573536be5c84204886a2fdbe4a41f85"
format_stamp: "Formatted at 2026-09-29 01:56:16 on dist-test-slave-vz23"
I20260929 01:56:16.629487 30620 ts_tablet_manager.cc:1962] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:56:16.643895 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:42695
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:35327,127.0.71.125:35679,127.0.71.124:39001
--webserver_port=42151
--webserver_interface=127.0.71.65
--time_source=system_unsync
--tserver_master_addrs=127.0.71.126:35327,127.0.71.125:35679,127.0.71.124:39001,127.0.71.123:34857 with env {}
I20260929 01:56:16.647212 30620 ts_tablet_manager.cc:1975] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 01:56:16.647490 30620 log.cc:1199] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260929 01:56:16.648094 30620 ts_tablet_manager.cc:1996] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85: Deleting consensus metadata
I20260929 01:56:16.648380 30620 master_runner.cc:316] Copying system tablet from 127.0.71.125:35679
I20260929 01:56:16.666808 30620 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85: tablet copy: Beginning tablet copy session from remote peer at address 127.0.71.125:35679
I20260929 01:56:16.686764 30339 tablet_copy_service.cc:140] P f8ed66573e594723a8f14f4794e3a138: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 6573536be5c84204886a2fdbe4a41f85 ({username='slave'} at 127.0.0.1:33120)
I20260929 01:56:16.687162 30339 tablet_copy_service.cc:161] P f8ed66573e594723a8f14f4794e3a138: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 6573536be5c84204886a2fdbe4a41f85 at {username='slave'} at 127.0.0.1:33120: session id = 6573536be5c84204886a2fdbe4a41f85-00000000000000000000000000000000
I20260929 01:56:16.692693 30339 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 01:56:16.696079 30620 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 01:56:16.704236 30620 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85: tablet copy: Starting download of 0 data blocks...
I20260929 01:56:16.704787 30620 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85: tablet copy: Starting download of 1 WAL segments...
I20260929 01:56:16.708839 30620 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 01:56:16.714628 30339 tablet_copy_service.cc:342] P f8ed66573e594723a8f14f4794e3a138: Request end of tablet copy session 6573536be5c84204886a2fdbe4a41f85-00000000000000000000000000000000 received from {username='slave'} at 127.0.0.1:33120
I20260929 01:56:16.715148 30339 tablet_copy_service.cc:434] P f8ed66573e594723a8f14f4794e3a138: ending tablet copy session 6573536be5c84204886a2fdbe4a41f85-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 6573536be5c84204886a2fdbe4a41f85
I20260929 01:56:16.731702 30620 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:16.735116 30734 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:16.736908 30735 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:16.746141 30737 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:16.748224 30620 server_base.cc:1061] running on GCE node
I20260929 01:56:16.748958 30620 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:56:16.749107 30620 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:56:16.749173 30620 hybrid_clock.cc:648] HybridClock initialized: now 1790646976749144 us; error 0 us; skew 500 ppm
I20260929 01:56:16.749457 30620 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:16.751257 30620 webserver.cc:533] Webserver started at http://127.0.71.123:36485/ using document root <none> and password file <none>
I20260929 01:56:16.751807 30620 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:16.751916 30620 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:16.755971 30620 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:56:16.758729 30742 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:16.759321 30620 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:56:16.759570 30620 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "6573536be5c84204886a2fdbe4a41f85"
format_stamp: "Formatted at 2026-09-29 01:56:16 on dist-test-slave-vz23"
I20260929 01:56:16.759913 30620 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:16.815099 30620 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:16.816218 30620 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:16.856668 30620 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.123:34857
I20260929 01:56:16.856721 30793 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.123:34857 every 8 connection(s)
I20260929 01:56:16.858174 30620 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
I20260929 01:56:16.865090 30794 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:56:16.869203 30794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85: Bootstrap starting.
I20260929 01:56:16.891140 30794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:56:16.891960 30794 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85: Bootstrap complete.
I20260929 01:56:16.893465 30794 raft_consensus.cc:359] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: true } }
I20260929 01:56:16.893796 30794 raft_consensus.cc:740] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 6573536be5c84204886a2fdbe4a41f85, State: Initialized, Role: LEARNER
I20260929 01:56:16.894111 30794 consensus_queue.cc:260] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: true } }
I20260929 01:56:16.895123 30797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: true } } }
I20260929 01:56:16.895468 30794 sys_catalog.cc:565] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:56:16.895620 30797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85 [sys.catalog]: This master's current role is: LEARNER
I20260929 01:56:16.909286 30808 catalog_manager.cc:1308] Loaded cluster ID: 28270ee6cb5d4aec85e53ccce360352e
I20260929 01:56:16.909435 30808 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85: loading cluster ID for follower catalog manager: success
I20260929 01:56:16.913936 30808 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85: acquiring CA information for follower catalog manager: success
I20260929 01:56:16.917240 30808 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:56:16.971086 30769 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260929 01:56:16.972182 30797 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8ed66573e594723a8f14f4794e3a138. Latest consensus state: current_term: 1 leader_uuid: "f8ed66573e594723a8f14f4794e3a138" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: true } } }
I20260929 01:56:16.972882 30797 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85 [sys.catalog]: This master's current role is: LEARNER
W20260929 01:56:17.182483 30729 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:17.182976 30729 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:17.183161 30729 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:17.196077 30729 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:17.196593 30729 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:56:17.211701 30729 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:42695
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=42151
--tserver_master_addrs=127.0.71.126:35327,127.0.71.125:35679,127.0.71.124:39001,127.0.71.123:34857
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:17.213956 30729 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:17.216290 30729 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:17.232226 30816 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:17.232229 30815 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:17.233531 30818 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:17.235030 30729 server_base.cc:1061] running on GCE node
I20260929 01:56:17.236297 30729 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:56:17.236799 30729 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:56:17.236948 30729 hybrid_clock.cc:648] HybridClock initialized: now 1790646977236915 us; error 0 us; skew 500 ppm
I20260929 01:56:17.237423 30729 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:17.242236 30729 webserver.cc:533] Webserver started at http://127.0.71.65:42151/ using document root <none> and password file <none>
I20260929 01:56:17.243280 30729 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:17.243422 30729 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:17.251080 30729 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:56:17.255555 30823 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:17.258244 30729 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260929 01:56:17.258522 30729 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "abb492f02115498487fc53f1ee17886f"
format_stamp: "Formatted at 2026-09-29 01:56:15 on dist-test-slave-vz23"
I20260929 01:56:17.259506 30729 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:17.281170 30729 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:17.282358 30729 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:17.282704 30729 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:17.284207 30729 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:56:17.286437 30729 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:56:17.286536 30729 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:17.286690 30729 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:56:17.286762 30729 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:17.338095 30729 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:42695
I20260929 01:56:17.338117 30935 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:42695 every 8 connection(s)
I20260929 01:56:17.340164 30729 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:56:17.345126 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 30729
I20260929 01:56:17.373373 30936 heartbeater.cc:344] Connected to a master server at 127.0.71.123:34857
I20260929 01:56:17.373963 30936 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:17.374298 30938 heartbeater.cc:344] Connected to a master server at 127.0.71.126:35327
I20260929 01:56:17.374480 30938 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:17.375393 30939 heartbeater.cc:344] Connected to a master server at 127.0.71.125:35679
I20260929 01:56:17.375685 30939 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:17.375933 30936 heartbeater.cc:507] Master 127.0.71.123:34857 requested a full tablet report, sending...
I20260929 01:56:17.376402 30939 heartbeater.cc:507] Master 127.0.71.125:35679 requested a full tablet report, sending...
I20260929 01:56:17.377321 30938 heartbeater.cc:507] Master 127.0.71.126:35327 requested a full tablet report, sending...
I20260929 01:56:17.379527 30249 ts_manager.cc:194] Re-registered known tserver with Master: abb492f02115498487fc53f1ee17886f (127.0.71.65:42695)
I20260929 01:56:17.379670 30759 ts_manager.cc:194] Registered new tserver with Master: abb492f02115498487fc53f1ee17886f (127.0.71.65:42695)
I20260929 01:56:17.380172 30937 heartbeater.cc:344] Connected to a master server at 127.0.71.124:39001
I20260929 01:56:17.380354 30937 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:17.381088 30937 heartbeater.cc:507] Master 127.0.71.124:39001 requested a full tablet report, sending...
I20260929 01:56:17.381068 30319 ts_manager.cc:194] Re-registered known tserver with Master: abb492f02115498487fc53f1ee17886f (127.0.71.65:42695)
I20260929 01:56:17.382301 30319 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:45005
I20260929 01:56:17.382753 30389 ts_manager.cc:194] Re-registered known tserver with Master: abb492f02115498487fc53f1ee17886f (127.0.71.65:42695)
I20260929 01:56:17.384204 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:56:17.403769 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:56:17.404515 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 30219
I20260929 01:56:17.406380 30279 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:17.501595 30810 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8ed66573e594723a8f14f4794e3a138" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: true } } }
I20260929 01:56:17.502027 30810 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:17.504537 30951 raft_consensus.cc:1064] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138: attempting to promote NON_VOTER 6573536be5c84204886a2fdbe4a41f85 to VOTER
I20260929 01:56:17.506022 30951 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: false } }
I20260929 01:56:17.510018 30769 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85 [term 1 LEARNER]: Refusing update from remote peer f8ed66573e594723a8f14f4794e3a138: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:56:17.510397 30398 raft_consensus.cc:1275] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d [term 1 FOLLOWER]: Refusing update from remote peer f8ed66573e594723a8f14f4794e3a138: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:56:17.511286 30953 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:56:17.511899 30471 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:56:17.518791 30471 raft_consensus.cc:2955] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6573536be5c84204886a2fdbe4a41f85 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: false } } }
I20260929 01:56:17.520004 30769 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6573536be5c84204886a2fdbe4a41f85 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: false } } }
I20260929 01:56:17.525386 30953 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "f8ed66573e594723a8f14f4794e3a138" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: false } } }
I20260929 01:56:17.524986 30955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8ed66573e594723a8f14f4794e3a138" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: false } } }
I20260929 01:56:17.525974 30953 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:17.526928 30471 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8ed66573e594723a8f14f4794e3a138" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: false } } }
I20260929 01:56:17.527000 30953 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8ed66573e594723a8f14f4794e3a138" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: false } } }
I20260929 01:56:17.527647 30471 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:17.527047 30398 raft_consensus.cc:2955] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6573536be5c84204886a2fdbe4a41f85 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: false } } }
I20260929 01:56:17.527676 30953 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:17.528532 30955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 6573536be5c84204886a2fdbe4a41f85 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:17.532953 30956 sys_catalog.cc:455] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8ed66573e594723a8f14f4794e3a138" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: false } } }
I20260929 01:56:17.533443 30956 sys_catalog.cc:458] T 00000000000000000000000000000000 P bf63a07c4759445a9c28ef8a8326165d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:17.556422 30259 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [term 1 FOLLOWER]: Refusing update from remote peer f8ed66573e594723a8f14f4794e3a138: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:56:17.557410 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 30219
I20260929 01:56:17.558132 30953 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [LEADER]: Connected to new peer: Peer: permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260929 01:56:17.589286 30304 connection.cc:573] client connection to 127.0.71.126:35327 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:56:17.590193 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:35327
--webserver_interface=127.0.71.126
--webserver_port=34393
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:35327,127.0.71.125:35679,127.0.71.124:39001
--master_addresses=127.0.71.126:35327,127.0.71.125:35679,127.0.71.124:39001,127.0.71.123:34857 with env {}
W20260929 01:56:17.592336 30304 consensus_peers.cc:597] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 -> Peer 7907493231354841a2732fbae6a131b4 (127.0.71.126:35327): Couldn't send request to peer 7907493231354841a2732fbae6a131b4. Status: Network error: Client connection negotiation failed: client connection to 127.0.71.126:35327: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260929 01:56:18.053448 30959 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:18.053903 30959 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:18.054049 30959 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:18.067689 30959 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:56:18.067850 30959 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:18.068004 30959 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:56:18.068097 30959 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:56:18.082402 30959 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:35327,127.0.71.125:35679,127.0.71.124:39001,127.0.71.123:34857
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:35327
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=34393
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:18.084764 30959 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:18.087352 30959 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:18.100510 30964 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:18.101646 30965 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:18.102016 30967 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:18.104430 30959 server_base.cc:1061] running on GCE node
I20260929 01:56:18.105746 30959 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:56:18.106125 30959 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:56:18.106288 30959 hybrid_clock.cc:648] HybridClock initialized: now 1790646978106235 us; error 0 us; skew 500 ppm
I20260929 01:56:18.107479 30959 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:18.112948 30959 webserver.cc:533] Webserver started at http://127.0.71.126:34393/ using document root <none> and password file <none>
I20260929 01:56:18.114023 30959 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:18.114209 30959 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:18.123600 30959 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.001s sys 0.004s
I20260929 01:56:18.128556 30974 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:18.131922 30959 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.001s sys 0.002s
I20260929 01:56:18.132733 30959 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "7907493231354841a2732fbae6a131b4"
format_stamp: "Formatted at 2026-09-29 01:56:13 on dist-test-slave-vz23"
I20260929 01:56:18.133911 30959 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:18.164700 30959 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:18.165756 30959 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:18.166062 30959 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:18.191443 30959 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:35327
I20260929 01:56:18.191478 31025 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:35327 every 8 connection(s)
I20260929 01:56:18.193601 30959 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:56:18.201223 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 30959
I20260929 01:56:18.201696 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 30729
I20260929 01:56:18.204257 31026 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:56:18.209944 31026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4: Bootstrap starting.
I20260929 01:56:18.214306 30931 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:18.234946 31026 log.cc:826] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4: Log is configured to *not* fsync() on all Append() calls
I20260929 01:56:18.249155 31026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:56:18.249918 31026 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4: Bootstrap complete.
I20260929 01:56:18.258689 31026 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: true } }
I20260929 01:56:18.259760 31026 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7907493231354841a2732fbae6a131b4, State: Initialized, Role: FOLLOWER
I20260929 01:56:18.260890 31026 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: true } }
I20260929 01:56:18.262385 31032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: true } } }
I20260929 01:56:18.262813 31032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:18.263705 31026 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:56:18.278077 31043 catalog_manager.cc:1308] Loaded cluster ID: 28270ee6cb5d4aec85e53ccce360352e
I20260929 01:56:18.280949 31043 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4: loading cluster ID for follower catalog manager: success
I20260929 01:56:18.286495 31043 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4: acquiring CA information for follower catalog manager: success
I20260929 01:56:18.291663 31043 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:56:18.345042 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 30729
I20260929 01:56:18.369643 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:42695
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:35327,127.0.71.125:35679,127.0.71.124:39001
--webserver_port=42151
--webserver_interface=127.0.71.65
--time_source=system_unsync
--tserver_master_addrs=127.0.71.126:35327,127.0.71.125:35679,127.0.71.124:39001,127.0.71.123:34857 with env {}
I20260929 01:56:18.549507 31001 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 6573536be5c84204886a2fdbe4a41f85 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: false } } }
I20260929 01:56:18.558941 31052 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "f8ed66573e594723a8f14f4794e3a138" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: false } } }
I20260929 01:56:18.559521 31052 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:18.559748 31032 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8ed66573e594723a8f14f4794e3a138. Latest consensus state: current_term: 1 leader_uuid: "f8ed66573e594723a8f14f4794e3a138" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: false } } }
I20260929 01:56:18.560040 31032 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7907493231354841a2732fbae6a131b4 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:56:18.827877 31050 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:18.828316 31050 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:18.828423 31050 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:18.839988 31050 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:18.840195 31050 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:56:18.851735 31050 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:42695
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=42151
--tserver_master_addrs=127.0.71.126:35327,127.0.71.125:35679,127.0.71.124:39001,127.0.71.123:34857
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:18.854275 31050 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:18.856827 31050 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:18.872242 31057 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:18.872217 31060 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:18.872314 31058 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:18.874248 31050 server_base.cc:1061] running on GCE node
I20260929 01:56:18.875486 31050 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:56:18.875798 31050 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:56:18.875969 31050 hybrid_clock.cc:648] HybridClock initialized: now 1790646978875902 us; error 0 us; skew 500 ppm
I20260929 01:56:18.876575 31050 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:18.880336 31050 webserver.cc:533] Webserver started at http://127.0.71.65:42151/ using document root <none> and password file <none>
I20260929 01:56:18.881547 31050 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:18.881681 31050 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:18.889348 31050 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.003s
I20260929 01:56:18.893844 31065 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:18.896998 31050 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.000s
I20260929 01:56:18.897270 31050 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "abb492f02115498487fc53f1ee17886f"
format_stamp: "Formatted at 2026-09-29 01:56:15 on dist-test-slave-vz23"
I20260929 01:56:18.898247 31050 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:18.923955 31050 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:18.925347 31050 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:18.925817 31050 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:18.927500 31050 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:56:18.930071 31050 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:56:18.930219 31050 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:18.930733 31050 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:56:18.930840 31050 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:18.936476 30953 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "f8ed66573e594723a8f14f4794e3a138" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "7907493231354841a2732fbae6a131b4" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35327 } } peers { permanent_uuid: "f8ed66573e594723a8f14f4794e3a138" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35679 } } peers { permanent_uuid: "bf63a07c4759445a9c28ef8a8326165d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 39001 } } peers { permanent_uuid: "6573536be5c84204886a2fdbe4a41f85" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 34857 } attrs { promote: false } } }
I20260929 01:56:18.936874 30953 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8ed66573e594723a8f14f4794e3a138 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:18.982486 31050 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:42695
I20260929 01:56:18.982542 31180 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:42695 every 8 connection(s)
I20260929 01:56:18.984575 31050 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:56:18.994334 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 31050
I20260929 01:56:18.995051 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 31050
I20260929 01:56:19.026172 31176 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:19.136075 31181 heartbeater.cc:344] Connected to a master server at 127.0.71.123:34857
I20260929 01:56:19.136883 31181 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:19.138482 31181 heartbeater.cc:507] Master 127.0.71.123:34857 requested a full tablet report, sending...
I20260929 01:56:19.139128 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 31050
I20260929 01:56:19.159536 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:42695
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:35327,127.0.71.125:35679,127.0.71.124:39001
--webserver_port=42151
--webserver_interface=127.0.71.65
--time_source=system_unsync
--tserver_master_addrs=127.0.71.126:35327,127.0.71.125:35679,127.0.71.124:39001,127.0.71.123:34857
--fault_before_start=1.0 with env {}
W20260929 01:56:19.159929 31194 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:56:19.013695 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:48153 (local address 127.0.71.125:35679)
0929 01:56:19.029446 (+ 15751us) server_negotiation.cc:207] Beginning negotiation
0929 01:56:19.029457 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0929 01:56:19.029490 (+ 33us) server_negotiation.cc:408] Connection header received
0929 01:56:19.029580 (+ 90us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:56:19.029586 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:56:19.029703 (+ 117us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:56:19.029883 (+ 180us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:56:19.144766 (+114883us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:56:19.145956 (+ 1190us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:56:19.158889 (+ 12933us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:48153: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":15577,"thread_start_us":15362,"threads_started":1}
W20260929 01:56:19.626075 31196 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:19.626461 31196 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:19.626633 31196 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:56:19.626725 31196 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:19.636430 31196 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:19.636749 31196 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:56:19.648233 31196 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:42695
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=42151
--tserver_master_addrs=127.0.71.126:35327,127.0.71.125:35679,127.0.71.124:39001,127.0.71.123:34857
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:19.650339 31196 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:19.652738 31196 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:19.668628 31203 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:19.670254 31202 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:19.672595 31205 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:19.673398 31196 server_base.cc:1061] running on GCE node
I20260929 01:56:19.674152 31196 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:56:19.674377 31196 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:56:19.674515 31196 hybrid_clock.cc:648] HybridClock initialized: now 1790646979674457 us; error 0 us; skew 500 ppm
I20260929 01:56:19.674902 31196 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:19.677911 31196 webserver.cc:533] Webserver started at http://127.0.71.65:42151/ using document root <none> and password file <none>
I20260929 01:56:19.678870 31196 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:19.679045 31196 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:19.686254 31196 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 01:56:19.689980 31210 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:19.691926 31196 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:56:19.692162 31196 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "abb492f02115498487fc53f1ee17886f"
format_stamp: "Formatted at 2026-09-29 01:56:15 on dist-test-slave-vz23"
I20260929 01:56:19.693153 31196 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:19.719667 31196 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:19.721478 31196 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:19.721899 31196 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:19.723613 31196 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:56:19.727101 31196 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:56:19.727283 31196 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:19.727512 31196 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:56:19.727612 31196 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:56:19.728421 31196 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:56:19.728600 31196 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:56:19.728689 31196 fault_injection.cc:59] injecting fault for kudu (pid 31196): FLAGS_fault_before_start (process will exit)
I20260929 01:56:19.753813 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:42695
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:35327,127.0.71.125:35679,127.0.71.124:39001
--webserver_port=42151
--webserver_interface=127.0.71.65
--time_source=system_unsync
--tserver_master_addrs=127.0.71.126:35327,127.0.71.125:35679,127.0.71.124:39001,127.0.71.123:34857 with env {}
W20260929 01:56:20.207301 31218 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:20.207723 31218 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:20.207864 31218 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:20.217041 31218 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:20.217293 31218 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:56:20.228040 31218 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:42695
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=42151
--tserver_master_addrs=127.0.71.126:35327,127.0.71.125:35679,127.0.71.124:39001,127.0.71.123:34857
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:20.230113 31218 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:20.232379 31218 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:20.247553 31226 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:20.247613 31223 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:20.247611 31224 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:20.248741 31218 server_base.cc:1061] running on GCE node
I20260929 01:56:20.249544 31218 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:56:20.249809 31218 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:56:20.249924 31218 hybrid_clock.cc:648] HybridClock initialized: now 1790646980249899 us; error 0 us; skew 500 ppm
I20260929 01:56:20.250309 31218 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:20.253990 31218 webserver.cc:533] Webserver started at http://127.0.71.65:42151/ using document root <none> and password file <none>
I20260929 01:56:20.255079 31218 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:20.255272 31218 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:20.265339 31218 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.007s sys 0.000s
I20260929 01:56:20.270848 31231 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:20.273064 31218 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.001s
I20260929 01:56:20.273402 31218 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "abb492f02115498487fc53f1ee17886f"
format_stamp: "Formatted at 2026-09-29 01:56:15 on dist-test-slave-vz23"
I20260929 01:56:20.274417 31218 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:20.309118 31218 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:20.310508 31218 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:20.311244 31218 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:20.313256 31218 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:56:20.315800 31218 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:56:20.315960 31218 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:20.316185 31218 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:56:20.316269 31218 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:20.365165 31218 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:42695
I20260929 01:56:20.365175 31343 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:42695 every 8 connection(s)
I20260929 01:56:20.366984 31218 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:56:20.370754 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 31218
I20260929 01:56:20.371207 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 31218
I20260929 01:56:20.395290 31344 heartbeater.cc:344] Connected to a master server at 127.0.71.123:34857
I20260929 01:56:20.395994 31344 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:20.396680 31339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:20.498333 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 31218
I20260929 01:56:20.519155 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 30959
W20260929 01:56:20.519758 31351 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:56:20.373231 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:56883 (local address 127.0.71.125:35679)
0929 01:56:20.378546 (+ 5315us) server_negotiation.cc:207] Beginning negotiation
0929 01:56:20.378556 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0929 01:56:20.381249 (+ 2693us) server_negotiation.cc:408] Connection header received
0929 01:56:20.382396 (+ 1147us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:56:20.382405 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:56:20.382506 (+ 101us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:56:20.392530 (+ 10024us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:56:20.392782 (+ 252us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:56:20.394685 (+ 1903us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:56:20.519336 (+124651us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:56883: BlockingRecv error: recv got EOF from 127.0.71.65:56883 (error 108)
Metrics: {"server-negotiator.queue_time_us":5160,"thread_start_us":5034,"threads_started":1}
W20260929 01:56:20.520669 31358 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:56:20.384003 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:38213 (local address 127.0.71.124:39001)
0929 01:56:20.397094 (+ 13091us) server_negotiation.cc:207] Beginning negotiation
0929 01:56:20.397105 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0929 01:56:20.397140 (+ 35us) server_negotiation.cc:408] Connection header received
0929 01:56:20.397243 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:56:20.397254 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:56:20.397411 (+ 157us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:56:20.397613 (+ 202us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:56:20.519020 (+121407us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:38213: BlockingRecv error: recv got EOF from 127.0.71.65:38213 (error 108)
Metrics: {"server-negotiator.queue_time_us":12883,"thread_start_us":12782,"threads_started":1}
W20260929 01:56:20.521660 31355 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:56:20.381612 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:49753 (local address 127.0.71.126:35327)
0929 01:56:20.382222 (+ 610us) server_negotiation.cc:207] Beginning negotiation
0929 01:56:20.382244 (+ 22us) server_negotiation.cc:400] Waiting for connection header
0929 01:56:20.390531 (+ 8287us) server_negotiation.cc:408] Connection header received
0929 01:56:20.390669 (+ 138us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:56:20.390680 (+ 11us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:56:20.390814 (+ 134us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:56:20.390967 (+ 153us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:56:20.397232 (+ 6265us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:56:20.399283 (+ 2051us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:56:20.519426 (+120143us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:49753: BlockingRecv error: recv got EOF from 127.0.71.65:49753 (error 108)
Metrics: {"server-negotiator.queue_time_us":454,"thread_start_us":515,"threads_started":1}
I20260929 01:56:20.530555 31021 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:20.662514 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 30959
I20260929 01:56:20.687515 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 30286
I20260929 01:56:20.689091 30349 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:20.812161 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 30286
I20260929 01:56:20.839097 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 30357
I20260929 01:56:20.840891 30419 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:20.962519 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 30357
I20260929 01:56:20.987741 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 30620
I20260929 01:56:20.990124 30789 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:21.127807 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 30620
[ OK ] ExternalMiniClusterTest.TestAddMaster/0 (8381 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/1
2026-09-29T01:56:21Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:56:21Z Disabled control of system clock
I20260929 01:56:21.213100 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:34931
--webserver_interface=127.0.71.126
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:34931,127.0.71.125:35733,127.0.71.124:37851 with env {}
W20260929 01:56:21.623499 31368 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:21.624037 31368 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:21.624120 31368 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:21.634832 31368 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:56:21.634955 31368 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:21.635021 31368 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:56:21.635077 31368 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:56:21.646989 31368 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:34931,127.0.71.125:35733,127.0.71.124:37851
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:34931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:21.648968 31368 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:21.651341 31368 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:21.663975 31373 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:21.664005 31374 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:21.665452 31376 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:21.666047 31368 server_base.cc:1061] running on GCE node
I20260929 01:56:21.667073 31368 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:21.669124 31368 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:21.670675 31368 hybrid_clock.cc:648] HybridClock initialized: now 1790646981670646 us; error 66 us; skew 500 ppm
I20260929 01:56:21.671186 31368 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:21.674499 31368 webserver.cc:533] Webserver started at http://127.0.71.126:40769/ using document root <none> and password file <none>
I20260929 01:56:21.675625 31368 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:21.675851 31368 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:21.676301 31368 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:56:21.679592 31368 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "12cdff735da147c59723d8561237671d"
format_stamp: "Formatted at 2026-09-29 01:56:21 on dist-test-slave-vz23"
I20260929 01:56:21.680822 31368 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "12cdff735da147c59723d8561237671d"
format_stamp: "Formatted at 2026-09-29 01:56:21 on dist-test-slave-vz23"
I20260929 01:56:21.688328 31368 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260929 01:56:21.693382 31382 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:21.695443 31368 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:56:21.695726 31368 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "12cdff735da147c59723d8561237671d"
format_stamp: "Formatted at 2026-09-29 01:56:21 on dist-test-slave-vz23"
I20260929 01:56:21.696038 31368 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:21.725282 31368 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:21.726430 31368 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:21.726807 31368 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:21.755796 31368 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:34931
I20260929 01:56:21.755843 31433 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:34931 every 8 connection(s)
I20260929 01:56:21.757895 31368 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:56:21.763927 31434 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:56:21.767696 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 31368
I20260929 01:56:21.768028 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:56:21.769580 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:35733
--webserver_interface=127.0.71.125
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:34931,127.0.71.125:35733,127.0.71.124:37851 with env {}
I20260929 01:56:21.774276 31434 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:21.805140 31434 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:56:21.807337 31385 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.125:35733: connect: Connection refused (error 111)
W20260929 01:56:21.810200 31434 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:35733: Network error: Client connection negotiation failed: client connection to 127.0.71.125:35733: connect: Connection refused (error 111)
I20260929 01:56:21.859902 31434 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } attempt: 1
W20260929 01:56:21.862691 31434 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:35733: Network error: Client connection negotiation failed: client connection to 127.0.71.125:35733: connect: Connection refused (error 111)
I20260929 01:56:21.931032 31434 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } attempt: 2
W20260929 01:56:21.934963 31434 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:35733: Network error: Client connection negotiation failed: client connection to 127.0.71.125:35733: connect: Connection refused (error 111)
I20260929 01:56:22.026463 31434 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } attempt: 3
W20260929 01:56:22.029220 31434 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:35733: Network error: Client connection negotiation failed: client connection to 127.0.71.125:35733: connect: Connection refused (error 111)
I20260929 01:56:22.172586 31434 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } attempt: 4
W20260929 01:56:22.175609 31434 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:35733: Network error: Client connection negotiation failed: client connection to 127.0.71.125:35733: connect: Connection refused (error 111)
W20260929 01:56:22.205178 31438 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:22.205688 31438 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:22.205837 31438 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:22.215888 31438 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:56:22.216104 31438 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:22.216233 31438 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:56:22.216336 31438 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:56:22.228384 31438 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:34931,127.0.71.125:35733,127.0.71.124:37851
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:35733
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:22.230578 31438 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:22.232963 31438 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:22.243809 31445 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:22.243777 31444 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:22.245517 31447 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:22.245689 31438 server_base.cc:1061] running on GCE node
I20260929 01:56:22.246687 31438 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:22.248672 31438 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:22.250017 31438 hybrid_clock.cc:648] HybridClock initialized: now 1790646982249956 us; error 91 us; skew 500 ppm
I20260929 01:56:22.250541 31438 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:22.254199 31438 webserver.cc:533] Webserver started at http://127.0.71.125:41523/ using document root <none> and password file <none>
I20260929 01:56:22.255321 31438 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:22.255517 31438 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:22.255973 31438 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:56:22.259243 31438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-1/data/instance:
uuid: "c1f706dc33d24fb291aa74a59b75f574"
format_stamp: "Formatted at 2026-09-29 01:56:22 on dist-test-slave-vz23"
I20260929 01:56:22.260499 31438 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-1/wal/instance:
uuid: "c1f706dc33d24fb291aa74a59b75f574"
format_stamp: "Formatted at 2026-09-29 01:56:22 on dist-test-slave-vz23"
I20260929 01:56:22.268333 31438 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260929 01:56:22.273456 31453 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:22.275439 31438 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:56:22.275741 31438 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "c1f706dc33d24fb291aa74a59b75f574"
format_stamp: "Formatted at 2026-09-29 01:56:22 on dist-test-slave-vz23"
I20260929 01:56:22.276047 31438 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:22.310597 31438 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:22.311736 31438 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:22.312172 31438 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:22.340095 31438 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:35733
I20260929 01:56:22.340112 31504 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:35733 every 8 connection(s)
I20260929 01:56:22.343227 31438 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:56:22.349202 31505 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:56:22.352944 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 31438
I20260929 01:56:22.353183 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:56:22.354619 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:37851
--webserver_interface=127.0.71.124
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:34931,127.0.71.125:35733,127.0.71.124:37851 with env {}
I20260929 01:56:22.360348 31505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:22.380126 31505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:22.394074 31505 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:56:22.395689 31456 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.124:37851: connect: Connection refused (error 111)
W20260929 01:56:22.398366 31505 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:37851: Network error: Client connection negotiation failed: client connection to 127.0.71.124:37851: connect: Connection refused (error 111)
I20260929 01:56:22.447919 31505 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } attempt: 1
W20260929 01:56:22.450702 31505 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:37851: Network error: Client connection negotiation failed: client connection to 127.0.71.124:37851: connect: Connection refused (error 111)
I20260929 01:56:22.475010 31434 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } attempt: 5
I20260929 01:56:22.483695 31434 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:56:22.486305 31434 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:37851: Network error: Client connection negotiation failed: client connection to 127.0.71.124:37851: connect: Connection refused (error 111)
I20260929 01:56:22.519099 31505 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } attempt: 2
W20260929 01:56:22.521586 31505 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:37851: Network error: Client connection negotiation failed: client connection to 127.0.71.124:37851: connect: Connection refused (error 111)
I20260929 01:56:22.537664 31434 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } attempt: 1
W20260929 01:56:22.540632 31434 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:37851: Network error: Client connection negotiation failed: client connection to 127.0.71.124:37851: connect: Connection refused (error 111)
I20260929 01:56:22.608952 31434 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } attempt: 2
W20260929 01:56:22.611305 31434 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:37851: Network error: Client connection negotiation failed: client connection to 127.0.71.124:37851: connect: Connection refused (error 111)
I20260929 01:56:22.612936 31505 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } attempt: 3
W20260929 01:56:22.615383 31505 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:37851: Network error: Client connection negotiation failed: client connection to 127.0.71.124:37851: connect: Connection refused (error 111)
I20260929 01:56:22.717644 31434 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } attempt: 3
W20260929 01:56:22.720304 31434 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:37851: Network error: Client connection negotiation failed: client connection to 127.0.71.124:37851: connect: Connection refused (error 111)
I20260929 01:56:22.758793 31505 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } attempt: 4
W20260929 01:56:22.762015 31505 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:37851: Network error: Client connection negotiation failed: client connection to 127.0.71.124:37851: connect: Connection refused (error 111)
W20260929 01:56:22.783461 31508 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:22.783898 31508 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:22.784060 31508 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:22.793485 31508 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:56:22.793704 31508 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:22.793813 31508 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:56:22.793902 31508 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:56:22.805477 31508 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:34931,127.0.71.125:35733,127.0.71.124:37851
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:37851
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:22.807463 31508 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:22.809911 31508 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:22.821275 31516 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:22.821280 31517 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:22.822898 31519 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:22.823859 31508 server_base.cc:1061] running on GCE node
I20260929 01:56:22.825557 31508 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:22.828037 31508 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:22.829470 31508 hybrid_clock.cc:648] HybridClock initialized: now 1790646982829399 us; error 63 us; skew 500 ppm
I20260929 01:56:22.830024 31508 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:22.833379 31508 webserver.cc:533] Webserver started at http://127.0.71.124:35575/ using document root <none> and password file <none>
I20260929 01:56:22.834462 31508 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:22.834596 31508 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:22.835011 31508 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:56:22.838080 31508 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-2/data/instance:
uuid: "7863c2a453d140e5b2093a08c0aa3415"
format_stamp: "Formatted at 2026-09-29 01:56:22 on dist-test-slave-vz23"
I20260929 01:56:22.839105 31508 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-2/wal/instance:
uuid: "7863c2a453d140e5b2093a08c0aa3415"
format_stamp: "Formatted at 2026-09-29 01:56:22 on dist-test-slave-vz23"
I20260929 01:56:22.846813 31508 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.003s
I20260929 01:56:22.852113 31525 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:22.854261 31508 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260929 01:56:22.854575 31508 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "7863c2a453d140e5b2093a08c0aa3415"
format_stamp: "Formatted at 2026-09-29 01:56:22 on dist-test-slave-vz23"
I20260929 01:56:22.854924 31508 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:22.878429 31508 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:22.879719 31508 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:22.880162 31508 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:22.897675 31434 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } attempt: 4
W20260929 01:56:22.900388 31434 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:37851: Network error: Client connection negotiation failed: client connection to 127.0.71.124:37851: connect: Connection refused (error 111)
I20260929 01:56:22.907020 31576 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:37851 every 8 connection(s)
I20260929 01:56:22.907035 31508 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:37851
I20260929 01:56:22.909198 31508 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:56:22.913483 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 31508
I20260929 01:56:22.913730 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:56:22.915305 31577 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:56:22.925299 31577 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:22.941864 31577 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:22.950695 31577 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:22.963513 31577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415: Bootstrap starting.
I20260929 01:56:22.967303 31577 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415: Neither blocks nor log segments found. Creating new log.
I20260929 01:56:22.968714 31577 log.cc:826] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415: Log is configured to *not* fsync() on all Append() calls
I20260929 01:56:22.972222 31577 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415: No bootstrap required, opened a new log
I20260929 01:56:22.978089 31577 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } }
I20260929 01:56:22.978541 31577 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:56:22.978699 31577 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7863c2a453d140e5b2093a08c0aa3415, State: Initialized, Role: FOLLOWER
I20260929 01:56:22.979485 31577 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } }
I20260929 01:56:22.980687 31584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } }
I20260929 01:56:22.981056 31584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:22.982204 31577 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:56:22.993552 31584 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:56:22.994033 31584 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } }
I20260929 01:56:22.996949 31584 leader_election.cc:290] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 12cdff735da147c59723d8561237671d (127.0.71.126:34931), c1f706dc33d24fb291aa74a59b75f574 (127.0.71.125:35733)
I20260929 01:56:22.991629 31508 master_runner.cc:187] Error getting master registration for 127.0.71.126:34931: OK, instance_id { permanent_uuid: "12cdff735da147c59723d8561237671d" instance_seqno: 1790646981748220 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:56:22.999506 31508 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:56:23.003152 31595 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:56:23.003345 31595 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:56:23.061271 31505 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } attempt: 5
I20260929 01:56:23.077672 31505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574: Bootstrap starting.
I20260929 01:56:23.085493 31505 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574: Neither blocks nor log segments found. Creating new log.
I20260929 01:56:23.087428 31505 log.cc:826] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574: Log is configured to *not* fsync() on all Append() calls
I20260929 01:56:23.092633 31505 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574: No bootstrap required, opened a new log
I20260929 01:56:23.103508 31505 raft_consensus.cc:359] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } }
I20260929 01:56:23.104017 31505 raft_consensus.cc:385] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:56:23.104156 31505 raft_consensus.cc:740] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1f706dc33d24fb291aa74a59b75f574, State: Initialized, Role: FOLLOWER
I20260929 01:56:23.105120 31505 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } }
I20260929 01:56:23.106668 31598 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } }
I20260929 01:56:23.107070 31598 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:23.108397 31505 sys_catalog.cc:565] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:56:23.108419 31480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7863c2a453d140e5b2093a08c0aa3415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1f706dc33d24fb291aa74a59b75f574" is_pre_election: true
I20260929 01:56:23.109167 31480 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 7863c2a453d140e5b2093a08c0aa3415 in term 0.
I20260929 01:56:23.110296 31528 leader_election.cc:304] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7863c2a453d140e5b2093a08c0aa3415, c1f706dc33d24fb291aa74a59b75f574; no voters:
I20260929 01:56:23.110954 31584 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:56:23.111156 31584 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:56:23.111287 31584 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:56:23.115315 31584 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } }
I20260929 01:56:23.119287 31584 leader_election.cc:290] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [CANDIDATE]: Term 1 election: Requested vote from peers 12cdff735da147c59723d8561237671d (127.0.71.126:34931), c1f706dc33d24fb291aa74a59b75f574 (127.0.71.125:35733)
I20260929 01:56:23.119655 31480 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7863c2a453d140e5b2093a08c0aa3415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c1f706dc33d24fb291aa74a59b75f574"
I20260929 01:56:23.119997 31480 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:56:23.123590 31480 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 7863c2a453d140e5b2093a08c0aa3415 in term 1.
I20260929 01:56:23.124606 31528 leader_election.cc:304] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 7863c2a453d140e5b2093a08c0aa3415, c1f706dc33d24fb291aa74a59b75f574; no voters:
I20260929 01:56:23.125160 31584 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:56:23.125836 31584 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [term 1 LEADER]: Becoming Leader. State: Replica: 7863c2a453d140e5b2093a08c0aa3415, State: Running, Role: LEADER
I20260929 01:56:23.126562 31584 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } }
I20260929 01:56:23.131270 31609 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7863c2a453d140e5b2093a08c0aa3415. Latest consensus state: current_term: 1 leader_uuid: "7863c2a453d140e5b2093a08c0aa3415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } }
I20260929 01:56:23.131716 31609 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:23.133417 31613 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:56:23.136512 31613 catalog_manager.cc:1533] Initializing Kudu cluster ID...
W20260929 01:56:23.136912 31531 tablet.cc:2433] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:56:23.131824 31438 master_runner.cc:187] Error getting master registration for 127.0.71.126:34931: OK, instance_id { permanent_uuid: "12cdff735da147c59723d8561237671d" instance_seqno: 1790646981748220 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:56:23.146529 31438 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:56:23.149451 31610 catalog_manager.cc:1607] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:56:23.149673 31610 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:56:23.163421 31480 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574 [term 1 FOLLOWER]: Refusing update from remote peer 7863c2a453d140e5b2093a08c0aa3415: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:56:23.165030 31609 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:56:23.177785 31434 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } attempt: 5
I20260929 01:56:23.188539 31598 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7863c2a453d140e5b2093a08c0aa3415. Latest consensus state: current_term: 1 leader_uuid: "7863c2a453d140e5b2093a08c0aa3415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } }
I20260929 01:56:23.188853 31598 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:23.197499 31614 mvcc.cc:204] Tried to move back new op lower bound from 7334490043021434880 to 7334490042892980224. Current Snapshot: MvccSnapshot[applied={T|T < 7334490043021434880}]
I20260929 01:56:23.202323 31584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7863c2a453d140e5b2093a08c0aa3415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } }
I20260929 01:56:23.202739 31584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:23.203212 31615 mvcc.cc:204] Tried to move back new op lower bound from 7334490043021434880 to 7334490042892980224. Current Snapshot: MvccSnapshot[applied={T|T < 7334490043021434880}]
I20260929 01:56:23.209080 31434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d: Bootstrap starting.
I20260929 01:56:23.214403 31434 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d: Neither blocks nor log segments found. Creating new log.
I20260929 01:56:23.215380 31613 catalog_manager.cc:1396] Generated new cluster ID: 5e7884bf68e04cc8ad4928f8636d89df
I20260929 01:56:23.215521 31613 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:56:23.220963 31598 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7863c2a453d140e5b2093a08c0aa3415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } }
I20260929 01:56:23.221299 31598 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:23.225435 31434 log.cc:826] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:56:23.236236 31434 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d: No bootstrap required, opened a new log
I20260929 01:56:23.252615 31434 raft_consensus.cc:359] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } }
I20260929 01:56:23.253299 31434 raft_consensus.cc:385] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:56:23.253525 31434 raft_consensus.cc:740] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12cdff735da147c59723d8561237671d, State: Initialized, Role: FOLLOWER
I20260929 01:56:23.254880 31434 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } }
I20260929 01:56:23.265632 31613 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 01:56:23.268303 31434 sys_catalog.cc:565] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:56:23.268925 31407 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:56:23.272994 31613 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:56:23.270645 31620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } }
I20260929 01:56:23.283975 31620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:23.283568 31408 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7863c2a453d140e5b2093a08c0aa3415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12cdff735da147c59723d8561237671d"
I20260929 01:56:23.283267 31409 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7863c2a453d140e5b2093a08c0aa3415" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "12cdff735da147c59723d8561237671d" is_pre_election: true
I20260929 01:56:23.289817 31407 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [term 1 FOLLOWER]: Refusing update from remote peer 7863c2a453d140e5b2093a08c0aa3415: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:56:23.291872 31609 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:56:23.315356 31613 catalog_manager.cc:6219] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415: Generated new TSK 0
I20260929 01:56:23.316836 31613 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:56:23.336475 31620 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7863c2a453d140e5b2093a08c0aa3415. Latest consensus state: current_term: 1 leader_uuid: "7863c2a453d140e5b2093a08c0aa3415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } }
I20260929 01:56:23.336491 31627 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7863c2a453d140e5b2093a08c0aa3415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } }
I20260929 01:56:23.336833 31620 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:23.336853 31627 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:23.338850 31584 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7863c2a453d140e5b2093a08c0aa3415" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } }
I20260929 01:56:23.339190 31584 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:23.345558 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:34931,127.0.71.125:35733,127.0.71.124:37851
--builtin_ntp_servers=127.0.71.84:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:56:23.432925 31637 catalog_manager.cc:1308] Loaded cluster ID: 5e7884bf68e04cc8ad4928f8636d89df
I20260929 01:56:23.433072 31637 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d: loading cluster ID for follower catalog manager: success
I20260929 01:56:23.438350 31637 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d: acquiring CA information for follower catalog manager: success
I20260929 01:56:23.450911 31637 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 01:56:23.984669 31634 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:23.985235 31634 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:23.985596 31634 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:23.997349 31634 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:23.997641 31634 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:56:24.010325 31634 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:34931,127.0.71.125:35733,127.0.71.124:37851
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:24.013835 31634 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:24.016417 31634 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:24.031113 31643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:24.032990 31644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:24.037619 31646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:24.038774 31634 server_base.cc:1061] running on GCE node
I20260929 01:56:24.040643 31634 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:24.042120 31634 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:24.043385 31634 hybrid_clock.cc:648] HybridClock initialized: now 1790646984043327 us; error 67 us; skew 500 ppm
I20260929 01:56:24.043886 31634 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:24.047341 31634 webserver.cc:533] Webserver started at http://127.0.71.65:36171/ using document root <none> and password file <none>
I20260929 01:56:24.048501 31634 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:24.048673 31634 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:24.049144 31634 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:56:24.052259 31634 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "c63823177542480d8d2708524259e2a1"
format_stamp: "Formatted at 2026-09-29 01:56:24 on dist-test-slave-vz23"
I20260929 01:56:24.053576 31634 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "c63823177542480d8d2708524259e2a1"
format_stamp: "Formatted at 2026-09-29 01:56:24 on dist-test-slave-vz23"
I20260929 01:56:24.061203 31634 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260929 01:56:24.066537 31652 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:24.068863 31634 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:56:24.069150 31634 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "c63823177542480d8d2708524259e2a1"
format_stamp: "Formatted at 2026-09-29 01:56:24 on dist-test-slave-vz23"
I20260929 01:56:24.069446 31634 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:24.100905 31634 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:24.101982 31634 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:24.102310 31634 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:24.103497 31634 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:56:24.105644 31634 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:56:24.105768 31634 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:24.105928 31634 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:56:24.106005 31634 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:24.150650 31634 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:39671
I20260929 01:56:24.150708 31764 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:39671 every 8 connection(s)
I20260929 01:56:24.152797 31634 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:56:24.153337 31610 catalog_manager.cc:1308] Loaded cluster ID: 5e7884bf68e04cc8ad4928f8636d89df
I20260929 01:56:24.153553 31610 catalog_manager.cc:1601] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574: loading cluster ID for follower catalog manager: success
I20260929 01:56:24.156677 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 31634
I20260929 01:56:24.156925 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
I20260929 01:56:24.161945 31610 catalog_manager.cc:1623] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574: acquiring CA information for follower catalog manager: success
I20260929 01:56:24.166785 31610 catalog_manager.cc:1651] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:56:24.193919 31765 heartbeater.cc:344] Connected to a master server at 127.0.71.124:37851
I20260929 01:56:24.194373 31765 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:24.194577 31767 heartbeater.cc:344] Connected to a master server at 127.0.71.125:35733
I20260929 01:56:24.194705 31767 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:24.195749 31765 heartbeater.cc:507] Master 127.0.71.124:37851 requested a full tablet report, sending...
I20260929 01:56:24.195825 31767 heartbeater.cc:507] Master 127.0.71.125:35733 requested a full tablet report, sending...
I20260929 01:56:24.198902 31541 ts_manager.cc:194] Registered new tserver with Master: c63823177542480d8d2708524259e2a1 (127.0.71.65:39671)
I20260929 01:56:24.199712 31470 ts_manager.cc:194] Registered new tserver with Master: c63823177542480d8d2708524259e2a1 (127.0.71.65:39671)
I20260929 01:56:24.201354 31541 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:54665
I20260929 01:56:24.202872 31766 heartbeater.cc:344] Connected to a master server at 127.0.71.126:34931
I20260929 01:56:24.203053 31766 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:24.203649 31766 heartbeater.cc:507] Master 127.0.71.126:34931 requested a full tablet report, sending...
I20260929 01:56:24.206512 31399 ts_manager.cc:194] Registered new tserver with Master: c63823177542480d8d2708524259e2a1 (127.0.71.65:39671)
I20260929 01:56:24.226107 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:56:24.228914 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.123:42335
--webserver_interface=127.0.71.123
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:34931,127.0.71.125:35733,127.0.71.124:37851,127.0.71.123:42335 with env {}
W20260929 01:56:24.654311 31775 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:24.654788 31775 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:24.654929 31775 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:24.664520 31775 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:56:24.664697 31775 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:24.664798 31775 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:56:24.664891 31775 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:56:24.676792 31775 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:34931,127.0.71.125:35733,127.0.71.124:37851,127.0.71.123:42335
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.123:42335
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.0.71.123
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:24.678557 31775 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:24.680938 31775 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:24.692132 31781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:24.692529 31784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:24.692992 31782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:24.693044 31775 server_base.cc:1061] running on GCE node
I20260929 01:56:24.694341 31775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:24.696163 31775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:24.697460 31775 hybrid_clock.cc:648] HybridClock initialized: now 1790646984697403 us; error 64 us; skew 500 ppm
I20260929 01:56:24.698021 31775 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:24.701000 31775 webserver.cc:533] Webserver started at http://127.0.71.123:44181/ using document root <none> and password file <none>
I20260929 01:56:24.701946 31775 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:24.702102 31775 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:24.702527 31775 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:56:24.705806 31775 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-3/data/instance:
uuid: "466a27cc49f54943ae6989c0370eb31c"
format_stamp: "Formatted at 2026-09-29 01:56:24 on dist-test-slave-vz23"
I20260929 01:56:24.706862 31775 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-3/wal/instance:
uuid: "466a27cc49f54943ae6989c0370eb31c"
format_stamp: "Formatted at 2026-09-29 01:56:24 on dist-test-slave-vz23"
I20260929 01:56:24.714809 31775 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260929 01:56:24.720685 31790 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:24.723074 31775 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260929 01:56:24.723331 31775 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "466a27cc49f54943ae6989c0370eb31c"
format_stamp: "Formatted at 2026-09-29 01:56:24 on dist-test-slave-vz23"
I20260929 01:56:24.723740 31775 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:24.751739 31775 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:24.752977 31775 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:24.753427 31775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:24.781906 31775 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.123:42335
I20260929 01:56:24.781917 31841 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.123:42335 every 8 connection(s)
I20260929 01:56:24.783898 31775 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
I20260929 01:56:24.784718 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 31775
I20260929 01:56:24.790982 31842 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:56:24.801633 31842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:24.818411 31842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:24.829766 31842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:24.841157 31842 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42335 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:24.854632 31842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c: Bootstrap starting.
I20260929 01:56:24.858891 31842 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c: Neither blocks nor log segments found. Creating new log.
I20260929 01:56:24.860759 31842 log.cc:826] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c: Log is configured to *not* fsync() on all Append() calls
I20260929 01:56:24.865191 31842 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c: No bootstrap required, opened a new log
I20260929 01:56:24.872020 31842 raft_consensus.cc:359] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42335 } }
I20260929 01:56:24.872483 31842 raft_consensus.cc:385] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:56:24.872629 31842 raft_consensus.cc:740] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 466a27cc49f54943ae6989c0370eb31c, State: Initialized, Role: FOLLOWER
I20260929 01:56:24.873333 31842 consensus_queue.cc:260] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42335 } }
I20260929 01:56:24.874358 31851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42335 } } }
I20260929 01:56:24.874755 31851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:24.875628 31842 sys_catalog.cc:565] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:56:24.888803 31862 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:56:24.889086 31862 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:56:24.900805 31775 master_runner.cc:429] Detected that this master 466a27cc49f54943ae6989c0370eb31c is joining an existing cluster
I20260929 01:56:24.900970 31775 catalog_manager.cc:1037] Shutting down the Catalog Manager
I20260929 01:56:24.902402 31775 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c [term 0 FOLLOWER]: Raft consensus shutting down.
I20260929 01:56:24.902951 31775 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c [term 0 FOLLOWER]: Raft consensus is shut down!
I20260929 01:56:24.903239 31775 tablet_replica.cc:333] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c: stopping tablet replica
I20260929 01:56:24.905469 31775 master_runner.cc:434] Initiating AddMaster RPC to add 127.0.71.123:42335
I20260929 01:56:24.953845 31541 catalog_manager.cc:7351] Initiating ChangeConfig request to add master 127.0.71.123:42335: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "466a27cc49f54943ae6989c0370eb31c"
member_type: NON_VOTER
last_known_addr {
host: "127.0.71.123"
port: 42335
}
attrs {
promote: true
}
}
dest_uuid: "7863c2a453d140e5b2093a08c0aa3415"
cas_config_opid_index: -1
I20260929 01:56:24.955536 31541 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: true } }
I20260929 01:56:24.960132 31407 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [term 1 FOLLOWER]: Refusing update from remote peer 7863c2a453d140e5b2093a08c0aa3415: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 01:56:24.960132 31480 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574 [term 1 FOLLOWER]: Refusing update from remote peer 7863c2a453d140e5b2093a08c0aa3415: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20260929 01:56:24.961318 31528 consensus_peers.cc:597] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 -> Peer 466a27cc49f54943ae6989c0370eb31c (127.0.71.123:42335): Couldn't send request to peer 466a27cc49f54943ae6989c0370eb31c. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260929 01:56:24.961684 31776 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:56:24.962220 31873 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:56:24.967141 31873 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 466a27cc49f54943ae6989c0370eb31c (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: true } } }
I20260929 01:56:24.968601 31480 raft_consensus.cc:2955] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 466a27cc49f54943ae6989c0370eb31c (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: true } } }
I20260929 01:56:24.969376 31407 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 466a27cc49f54943ae6989c0370eb31c (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: true } } }
I20260929 01:56:24.973889 31873 catalog_manager.cc:7355] Successfully completed master ChangeConfig request to add master 127.0.71.123:42335
I20260929 01:56:24.975261 31775 master.cc:562] Master@127.0.71.123:42335 shutting down...
I20260929 01:56:24.977190 31877 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7863c2a453d140e5b2093a08c0aa3415" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: true } } }
I20260929 01:56:24.977200 31878 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7863c2a453d140e5b2093a08c0aa3415" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: true } } }
I20260929 01:56:24.977608 31877 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:24.977649 31878 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:24.978346 31776 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7863c2a453d140e5b2093a08c0aa3415" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: true } } }
I20260929 01:56:24.978561 31873 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7863c2a453d140e5b2093a08c0aa3415" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: true } } }
I20260929 01:56:24.978746 31776 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:24.978787 31873 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:24.980696 31876 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7863c2a453d140e5b2093a08c0aa3415" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: true } } }
I20260929 01:56:24.981375 31876 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:25.008502 31775 master.cc:584] Master@127.0.71.123:42335 shutdown complete.
I20260929 01:56:25.008666 31775 master_runner.cc:306] Clearing existing system tablet
I20260929 01:56:25.009285 31775 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:25.009455 31775 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:25.013402 31775 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:56:25.015695 31883 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:25.016247 31775 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:25.016595 31775 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260929 01:56:25.016721 31775 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "466a27cc49f54943ae6989c0370eb31c"
format_stamp: "Formatted at 2026-09-29 01:56:24 on dist-test-slave-vz23"
I20260929 01:56:25.022046 31775 ts_tablet_manager.cc:1962] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:56:25.030072 31775 ts_tablet_manager.cc:1975] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 01:56:25.030292 31775 log.cc:1199] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260929 01:56:25.030809 31775 ts_tablet_manager.cc:1996] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c: Deleting consensus metadata
I20260929 01:56:25.031131 31775 master_runner.cc:316] Copying system tablet from 127.0.71.124:37851
I20260929 01:56:25.034408 31775 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c: tablet copy: Beginning tablet copy session from remote peer at address 127.0.71.124:37851
I20260929 01:56:25.043836 31562 tablet_copy_service.cc:140] P 7863c2a453d140e5b2093a08c0aa3415: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 466a27cc49f54943ae6989c0370eb31c ({username='slave'} at 127.0.0.1:45962)
I20260929 01:56:25.044189 31562 tablet_copy_service.cc:161] P 7863c2a453d140e5b2093a08c0aa3415: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 466a27cc49f54943ae6989c0370eb31c at {username='slave'} at 127.0.0.1:45962: session id = 466a27cc49f54943ae6989c0370eb31c-00000000000000000000000000000000
I20260929 01:56:25.049062 31562 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 01:56:25.052227 31775 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 01:56:25.060221 31775 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c: tablet copy: Starting download of 0 data blocks...
I20260929 01:56:25.060642 31775 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c: tablet copy: Starting download of 1 WAL segments...
I20260929 01:56:25.064188 31775 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 01:56:25.070034 31562 tablet_copy_service.cc:342] P 7863c2a453d140e5b2093a08c0aa3415: Request end of tablet copy session 466a27cc49f54943ae6989c0370eb31c-00000000000000000000000000000000 received from {username='slave'} at 127.0.0.1:45962
I20260929 01:56:25.070364 31562 tablet_copy_service.cc:434] P 7863c2a453d140e5b2093a08c0aa3415: ending tablet copy session 466a27cc49f54943ae6989c0370eb31c-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 466a27cc49f54943ae6989c0370eb31c
W20260929 01:56:25.078263 31775 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260929 01:56:25.083477 31775 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:25.087255 31890 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:25.087507 31891 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:25.089221 31893 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:25.089856 31775 server_base.cc:1061] running on GCE node
I20260929 01:56:25.090310 31775 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:25.091097 31775 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:25.092376 31775 hybrid_clock.cc:648] HybridClock initialized: now 1790646985092303 us; error 48 us; skew 500 ppm
I20260929 01:56:25.092748 31775 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:25.095259 31775 webserver.cc:533] Webserver started at http://127.0.71.123:34991/ using document root <none> and password file <none>
I20260929 01:56:25.095808 31775 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:25.095938 31775 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:25.099328 31775 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:56:25.101927 31899 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:25.102634 31775 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:56:25.102804 31775 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "466a27cc49f54943ae6989c0370eb31c"
format_stamp: "Formatted at 2026-09-29 01:56:24 on dist-test-slave-vz23"
I20260929 01:56:25.103016 31775 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:25.118916 31775 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:25.119817 31775 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:25.142179 31775 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.123:42335
I20260929 01:56:25.142197 31950 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.123:42335 every 8 connection(s)
I20260929 01:56:25.143491 31775 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
I20260929 01:56:25.148217 31951 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:56:25.151543 31951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c: Bootstrap starting.
I20260929 01:56:25.171734 31951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:56:25.172569 31951 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c: Bootstrap complete.
I20260929 01:56:25.174118 31951 raft_consensus.cc:359] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: true } }
I20260929 01:56:25.174463 31951 raft_consensus.cc:740] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 466a27cc49f54943ae6989c0370eb31c, State: Initialized, Role: LEARNER
I20260929 01:56:25.174839 31951 consensus_queue.cc:260] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: true } }
I20260929 01:56:25.175961 31955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: true } } }
I20260929 01:56:25.176548 31955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c [sys.catalog]: This master's current role is: LEARNER
I20260929 01:56:25.176574 31951 sys_catalog.cc:565] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:56:25.188282 31966 catalog_manager.cc:1308] Loaded cluster ID: 5e7884bf68e04cc8ad4928f8636d89df
I20260929 01:56:25.188412 31966 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c: loading cluster ID for follower catalog manager: success
I20260929 01:56:25.191892 31966 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c: acquiring CA information for follower catalog manager: success
I20260929 01:56:25.194669 31966 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:56:25.205736 31765 heartbeater.cc:499] Master 127.0.71.124:37851 was elected leader, sending a full tablet report...
I20260929 01:56:25.233870 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 31634
I20260929 01:56:25.243762 31760 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:25.344822 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 31634
I20260929 01:56:25.371723 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:39671
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:34931,127.0.71.125:35733,127.0.71.124:37851
--webserver_port=36171
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:34931,127.0.71.125:35733,127.0.71.124:37851,127.0.71.123:42335 with env {}
I20260929 01:56:25.440985 31926 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260929 01:56:25.442399 31955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7863c2a453d140e5b2093a08c0aa3415. Latest consensus state: current_term: 1 leader_uuid: "7863c2a453d140e5b2093a08c0aa3415" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: true } } }
I20260929 01:56:25.442768 31955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c [sys.catalog]: This master's current role is: LEARNER
W20260929 01:56:25.823959 31969 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:25.824462 31969 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:25.824680 31969 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:25.835006 31969 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:25.835335 31969 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:56:25.846930 31969 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:39671
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=36171
--tserver_master_addrs=127.0.71.126:34931,127.0.71.125:35733,127.0.71.124:37851,127.0.71.123:42335
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:25.849156 31969 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:25.851564 31969 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:25.868083 31977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:25.868132 31974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:25.868840 31975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:25.870388 31969 server_base.cc:1061] running on GCE node
I20260929 01:56:25.871243 31969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:25.872843 31969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:25.876720 31969 hybrid_clock.cc:648] HybridClock initialized: now 1790646985876313 us; error 99 us; skew 500 ppm
I20260929 01:56:25.877614 31969 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:25.881621 31969 webserver.cc:533] Webserver started at http://127.0.71.65:36171/ using document root <none> and password file <none>
I20260929 01:56:25.882710 31969 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:25.882925 31969 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:25.891340 31969 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.004s sys 0.000s
I20260929 01:56:25.892262 31879 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7863c2a453d140e5b2093a08c0aa3415" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: true } } }
I20260929 01:56:25.892572 31879 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:25.894256 31984 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415: attempting to promote NON_VOTER 466a27cc49f54943ae6989c0370eb31c to VOTER
I20260929 01:56:25.895320 31984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: false } }
I20260929 01:56:25.895955 31986 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:25.898034 31480 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574 [term 1 FOLLOWER]: Refusing update from remote peer 7863c2a453d140e5b2093a08c0aa3415: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:56:25.898118 31926 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c [term 1 LEARNER]: Refusing update from remote peer 7863c2a453d140e5b2093a08c0aa3415: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:56:25.898200 31407 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [term 1 FOLLOWER]: Refusing update from remote peer 7863c2a453d140e5b2093a08c0aa3415: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:56:25.898988 31969 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260929 01:56:25.898900 31879 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:56:25.899293 31987 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:56:25.899823 31984 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [LEADER]: Connected to new peer: Peer: permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:56:25.899209 31969 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "c63823177542480d8d2708524259e2a1"
format_stamp: "Formatted at 2026-09-29 01:56:24 on dist-test-slave-vz23"
I20260929 01:56:25.900341 31969 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:25.905059 31984 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 466a27cc49f54943ae6989c0370eb31c (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: false } } }
I20260929 01:56:25.905792 31407 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 466a27cc49f54943ae6989c0370eb31c (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: false } } }
I20260929 01:56:25.906026 31926 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 466a27cc49f54943ae6989c0370eb31c (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: false } } }
I20260929 01:56:25.910873 31983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "7863c2a453d140e5b2093a08c0aa3415" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: false } } }
I20260929 01:56:25.911317 31983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:25.911798 31983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7863c2a453d140e5b2093a08c0aa3415" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: false } } }
I20260929 01:56:25.912092 31983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:25.913125 31983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7863c2a453d140e5b2093a08c0aa3415" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: false } } }
I20260929 01:56:25.913477 31983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:25.913957 31984 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7863c2a453d140e5b2093a08c0aa3415" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: false } } }
I20260929 01:56:25.914289 31984 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:25.914280 31480 raft_consensus.cc:2955] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 466a27cc49f54943ae6989c0370eb31c (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: false } } }
I20260929 01:56:25.925139 31955 sys_catalog.cc:455] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7863c2a453d140e5b2093a08c0aa3415" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: false } } }
I20260929 01:56:25.925568 31955 sys_catalog.cc:458] T 00000000000000000000000000000000 P 466a27cc49f54943ae6989c0370eb31c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:25.929088 31989 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7863c2a453d140e5b2093a08c0aa3415" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: false } } }
I20260929 01:56:25.929863 31989 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:25.930614 31991 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "7863c2a453d140e5b2093a08c0aa3415" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: false } } }
I20260929 01:56:25.931063 31991 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1f706dc33d24fb291aa74a59b75f574 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:25.946372 31969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:25.947300 31969 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:25.947606 31969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:25.948819 31969 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:56:25.950873 31969 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:56:25.950968 31969 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:25.951114 31969 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:56:25.951184 31969 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:25.999454 31969 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:39671
I20260929 01:56:25.999485 32102 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:39671 every 8 connection(s)
I20260929 01:56:26.001673 31969 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:56:26.003463 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 31969
I20260929 01:56:26.030746 32104 heartbeater.cc:344] Connected to a master server at 127.0.71.124:37851
I20260929 01:56:26.030835 32106 heartbeater.cc:344] Connected to a master server at 127.0.71.125:35733
I20260929 01:56:26.030923 32105 heartbeater.cc:344] Connected to a master server at 127.0.71.126:34931
I20260929 01:56:26.031337 32106 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:26.031342 32105 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:26.031337 32104 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:26.031965 32103 heartbeater.cc:344] Connected to a master server at 127.0.71.123:42335
I20260929 01:56:26.032168 32103 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:26.033227 32105 heartbeater.cc:507] Master 127.0.71.126:34931 requested a full tablet report, sending...
I20260929 01:56:26.033265 32103 heartbeater.cc:507] Master 127.0.71.123:42335 requested a full tablet report, sending...
I20260929 01:56:26.033335 32106 heartbeater.cc:507] Master 127.0.71.125:35733 requested a full tablet report, sending...
I20260929 01:56:26.033365 32104 heartbeater.cc:507] Master 127.0.71.124:37851 requested a full tablet report, sending...
I20260929 01:56:26.036160 31470 ts_manager.cc:194] Re-registered known tserver with Master: c63823177542480d8d2708524259e2a1 (127.0.71.65:39671)
I20260929 01:56:26.036157 31399 ts_manager.cc:194] Re-registered known tserver with Master: c63823177542480d8d2708524259e2a1 (127.0.71.65:39671)
I20260929 01:56:26.036330 31541 ts_manager.cc:194] Re-registered known tserver with Master: c63823177542480d8d2708524259e2a1 (127.0.71.65:39671)
I20260929 01:56:26.037494 31541 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:50885
I20260929 01:56:26.038496 31916 ts_manager.cc:194] Registered new tserver with Master: c63823177542480d8d2708524259e2a1 (127.0.71.65:39671)
I20260929 01:56:26.040087 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:56:26.053898 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:56:26.054131 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 31368
I20260929 01:56:26.055419 31429 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:26.187631 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 31368
I20260929 01:56:26.212999 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:34931
--webserver_interface=127.0.71.126
--webserver_port=40769
--builtin_ntp_servers=127.0.71.84:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:34931,127.0.71.125:35733,127.0.71.124:37851
--master_addresses=127.0.71.126:34931,127.0.71.125:35733,127.0.71.124:37851,127.0.71.123:42335 with env {}
W20260929 01:56:26.331700 31528 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.126:34931: connect: Connection refused (error 111)
W20260929 01:56:26.333940 31528 consensus_peers.cc:597] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 -> Peer 12cdff735da147c59723d8561237671d (127.0.71.126:34931): Couldn't send request to peer 12cdff735da147c59723d8561237671d. Status: Network error: Client connection negotiation failed: client connection to 127.0.71.126:34931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260929 01:56:26.640733 32117 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:26.641167 32117 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:26.641340 32117 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:26.650930 32117 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:56:26.651105 32117 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:26.651216 32117 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:56:26.651312 32117 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:56:26.663327 32117 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:34931,127.0.71.125:35733,127.0.71.124:37851,127.0.71.123:42335
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:34931
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=40769
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:26.665403 32117 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:26.667634 32117 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:26.682708 32124 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:26.685989 32125 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:26.687134 32127 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:26.687896 32117 server_base.cc:1061] running on GCE node
I20260929 01:56:26.689858 32117 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:26.692780 32117 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:26.694267 32117 hybrid_clock.cc:648] HybridClock initialized: now 1790646986694188 us; error 74 us; skew 500 ppm
I20260929 01:56:26.694885 32117 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:26.699002 32117 webserver.cc:533] Webserver started at http://127.0.71.126:40769/ using document root <none> and password file <none>
I20260929 01:56:26.700350 32117 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:26.700616 32117 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:26.710928 32117 fs_manager.cc:714] Time spent opening directory manager: real 0.007s user 0.003s sys 0.005s
I20260929 01:56:26.716671 32133 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:26.719377 32117 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.001s
I20260929 01:56:26.719705 32117 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "12cdff735da147c59723d8561237671d"
format_stamp: "Formatted at 2026-09-29 01:56:21 on dist-test-slave-vz23"
I20260929 01:56:26.721272 32117 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:26.744277 32117 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:26.745560 32117 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:26.745898 32117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:26.773644 32117 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:34931
I20260929 01:56:26.773666 32184 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:34931 every 8 connection(s)
I20260929 01:56:26.775585 32117 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:56:26.776248 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 32117
I20260929 01:56:26.776676 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 31969
I20260929 01:56:26.787108 32185 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:56:26.788704 32098 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:26.793787 32185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d: Bootstrap starting.
I20260929 01:56:26.814656 32185 log.cc:826] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:56:26.828125 32185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:56:26.828995 32185 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d: Bootstrap complete.
I20260929 01:56:26.837675 32185 raft_consensus.cc:359] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: false } }
I20260929 01:56:26.838630 32185 raft_consensus.cc:740] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 12cdff735da147c59723d8561237671d, State: Initialized, Role: FOLLOWER
I20260929 01:56:26.839641 32185 consensus_queue.cc:260] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: false } }
I20260929 01:56:26.840830 32191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: false } } }
I20260929 01:56:26.841221 32191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:26.842271 32185 sys_catalog.cc:565] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:56:26.856055 32202 catalog_manager.cc:1308] Loaded cluster ID: 5e7884bf68e04cc8ad4928f8636d89df
I20260929 01:56:26.856256 32202 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d: loading cluster ID for follower catalog manager: success
I20260929 01:56:26.864609 32202 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d: acquiring CA information for follower catalog manager: success
I20260929 01:56:26.868383 32202 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:56:26.907084 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 31969
I20260929 01:56:26.929231 31983 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7863c2a453d140e5b2093a08c0aa3415" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: false } } }
I20260929 01:56:26.932576 31983 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:26.936278 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:39671
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:34931,127.0.71.125:35733,127.0.71.124:37851
--webserver_port=36171
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:34931,127.0.71.125:35733,127.0.71.124:37851,127.0.71.123:42335 with env {}
I20260929 01:56:26.948228 32191 sys_catalog.cc:455] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7863c2a453d140e5b2093a08c0aa3415. Latest consensus state: current_term: 1 leader_uuid: "7863c2a453d140e5b2093a08c0aa3415" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: false } } }
I20260929 01:56:26.948767 32191 sys_catalog.cc:458] T 00000000000000000000000000000000 P 12cdff735da147c59723d8561237671d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:27.336722 32208 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "7863c2a453d140e5b2093a08c0aa3415" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "12cdff735da147c59723d8561237671d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34931 } } peers { permanent_uuid: "c1f706dc33d24fb291aa74a59b75f574" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 35733 } } peers { permanent_uuid: "7863c2a453d140e5b2093a08c0aa3415" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 37851 } } peers { permanent_uuid: "466a27cc49f54943ae6989c0370eb31c" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42335 } attrs { promote: false } } }
I20260929 01:56:27.338428 32208 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 [sys.catalog]: This master's current role is: LEADER
W20260929 01:56:27.389199 32213 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:27.389688 32213 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:27.389881 32213 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:27.400053 32213 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:27.400377 32213 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:56:27.413130 32213 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:39671
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=36171
--tserver_master_addrs=127.0.71.126:34931,127.0.71.125:35733,127.0.71.124:37851,127.0.71.123:42335
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:27.415141 32213 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:27.417745 32213 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:27.432775 32218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:27.433158 32219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:27.434751 32221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:27.436333 32213 server_base.cc:1061] running on GCE node
I20260929 01:56:27.437160 32213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:27.438623 32213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:27.440172 32213 hybrid_clock.cc:648] HybridClock initialized: now 1790646987439986 us; error 157 us; skew 500 ppm
I20260929 01:56:27.441041 32213 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:27.445817 32213 webserver.cc:533] Webserver started at http://127.0.71.65:36171/ using document root <none> and password file <none>
I20260929 01:56:27.447309 32213 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:27.447530 32213 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:27.457551 32213 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 01:56:27.463083 32227 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:27.465660 32213 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.000s
I20260929 01:56:27.465963 32213 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "c63823177542480d8d2708524259e2a1"
format_stamp: "Formatted at 2026-09-29 01:56:24 on dist-test-slave-vz23"
I20260929 01:56:27.466917 32213 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:27.495499 32213 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:27.496826 32213 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:27.497184 32213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:27.498483 32213 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:56:27.501596 32213 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:56:27.501801 32213 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:27.502000 32213 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:56:27.502113 32213 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:27.551112 32213 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:39671
I20260929 01:56:27.551151 32339 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:39671 every 8 connection(s)
I20260929 01:56:27.553165 32213 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:56:27.557647 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 32213
I20260929 01:56:27.557991 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 32213
I20260929 01:56:27.583833 32340 heartbeater.cc:344] Connected to a master server at 127.0.71.123:42335
I20260929 01:56:27.584321 32340 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:27.585168 32341 heartbeater.cc:344] Connected to a master server at 127.0.71.124:37851
I20260929 01:56:27.585425 32341 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:27.585664 32343 heartbeater.cc:344] Connected to a master server at 127.0.71.125:35733
I20260929 01:56:27.585915 32343 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:27.586124 32340 heartbeater.cc:507] Master 127.0.71.123:42335 requested a full tablet report, sending...
I20260929 01:56:27.586184 32341 heartbeater.cc:507] Master 127.0.71.124:37851 requested a full tablet report, sending...
I20260929 01:56:27.586561 32343 heartbeater.cc:507] Master 127.0.71.125:35733 requested a full tablet report, sending...
I20260929 01:56:27.588496 31541 ts_manager.cc:194] Re-registered known tserver with Master: c63823177542480d8d2708524259e2a1 (127.0.71.65:39671)
I20260929 01:56:27.588721 31470 ts_manager.cc:194] Re-registered known tserver with Master: c63823177542480d8d2708524259e2a1 (127.0.71.65:39671)
I20260929 01:56:27.588903 31915 ts_manager.cc:194] Re-registered known tserver with Master: c63823177542480d8d2708524259e2a1 (127.0.71.65:39671)
I20260929 01:56:27.589771 31541 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:52051
I20260929 01:56:27.596853 32335 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:27.697449 32342 heartbeater.cc:344] Connected to a master server at 127.0.71.126:34931
I20260929 01:56:27.697860 32342 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:27.698472 32342 heartbeater.cc:507] Master 127.0.71.126:34931 requested a full tablet report, sending...
I20260929 01:56:27.698895 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 32213
I20260929 01:56:27.700937 32150 ts_manager.cc:194] Registered new tserver with Master: c63823177542480d8d2708524259e2a1 (127.0.71.65:39671)
W20260929 01:56:27.719458 32136 connection.cc:573] server connection from 127.0.71.65:52791 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:56:27.720516 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:39671
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:34931,127.0.71.125:35733,127.0.71.124:37851
--webserver_port=36171
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:34931,127.0.71.125:35733,127.0.71.124:37851,127.0.71.123:42335
--fault_before_start=1.0 with env {}
W20260929 01:56:28.195402 32356 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:28.195888 32356 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:28.196064 32356 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:56:28.196156 32356 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:28.206705 32356 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:28.207031 32356 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:56:28.222384 32356 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:39671
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=36171
--tserver_master_addrs=127.0.71.126:34931,127.0.71.125:35733,127.0.71.124:37851,127.0.71.123:42335
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:28.224562 32356 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:28.228150 32356 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:28.242798 32363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:28.242831 32364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:28.243571 32366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:28.244117 32356 server_base.cc:1061] running on GCE node
I20260929 01:56:28.244943 32356 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:28.246371 32356 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:28.247692 32356 hybrid_clock.cc:648] HybridClock initialized: now 1790646988247599 us; error 80 us; skew 500 ppm
I20260929 01:56:28.248122 32356 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:28.252171 32356 webserver.cc:533] Webserver started at http://127.0.71.65:36171/ using document root <none> and password file <none>
I20260929 01:56:28.253481 32356 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:28.253682 32356 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:28.261389 32356 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 01:56:28.266762 32372 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:28.268790 32356 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.003s
I20260929 01:56:28.269032 32356 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "c63823177542480d8d2708524259e2a1"
format_stamp: "Formatted at 2026-09-29 01:56:24 on dist-test-slave-vz23"
I20260929 01:56:28.269934 32356 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:28.285382 32356 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:28.286609 32356 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:28.287043 32356 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:28.288710 32356 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:56:28.290875 32356 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:56:28.291042 32356 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:28.291193 32356 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:56:28.291286 32356 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:56:28.291785 32356 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:56:28.291934 32356 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:56:28.292032 32356 fault_injection.cc:59] injecting fault for kudu (pid 32356): FLAGS_fault_before_start (process will exit)
I20260929 01:56:28.309963 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:39671
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:34931,127.0.71.125:35733,127.0.71.124:37851
--webserver_port=36171
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:33599
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:34931,127.0.71.125:35733,127.0.71.124:37851,127.0.71.123:42335 with env {}
W20260929 01:56:28.733354 32380 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:28.733978 32380 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:28.734210 32380 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:28.746142 32380 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:28.746673 32380 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:56:28.760330 32380 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:33599
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:39671
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=36171
--tserver_master_addrs=127.0.71.126:34931,127.0.71.125:35733,127.0.71.124:37851,127.0.71.123:42335
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:28.763501 32380 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:28.766203 32380 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:28.781994 32385 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:28.781327 32386 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:28.786751 32388 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:28.788321 32380 server_base.cc:1061] running on GCE node
I20260929 01:56:28.789388 32380 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:28.790833 32380 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:28.792678 32380 hybrid_clock.cc:648] HybridClock initialized: now 1790646988792577 us; error 70 us; skew 500 ppm
I20260929 01:56:28.793195 32380 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:28.796734 32380 webserver.cc:533] Webserver started at http://127.0.71.65:36171/ using document root <none> and password file <none>
I20260929 01:56:28.797825 32380 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:28.798060 32380 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:28.805857 32380 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 01:56:28.809762 32394 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:28.811784 32380 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:56:28.812055 32380 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "c63823177542480d8d2708524259e2a1"
format_stamp: "Formatted at 2026-09-29 01:56:24 on dist-test-slave-vz23"
I20260929 01:56:28.813210 32380 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:28.850701 32380 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:28.851807 32380 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:28.852164 32380 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:28.853371 32380 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:56:28.855611 32380 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:56:28.855763 32380 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:28.855942 32380 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:56:28.856021 32380 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:28.906011 32380 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:39671
I20260929 01:56:28.906095 32506 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:39671 every 8 connection(s)
I20260929 01:56:28.908751 32380 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:56:28.918499 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 32380
I20260929 01:56:28.918865 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 32380
I20260929 01:56:28.958570 32502 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:29.066401 32508 heartbeater.cc:344] Connected to a master server at 127.0.71.124:37851
I20260929 01:56:29.066958 32507 heartbeater.cc:344] Connected to a master server at 127.0.71.123:42335
I20260929 01:56:29.067466 32507 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:29.067462 32508 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:29.068202 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 32380
I20260929 01:56:29.088708 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 32117
W20260929 01:56:29.089378 32520 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:56:28.947364 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:41693 (local address 127.0.71.125:35733)
0929 01:56:28.949046 (+ 1682us) server_negotiation.cc:207] Beginning negotiation
0929 01:56:28.949054 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0929 01:56:28.955824 (+ 6770us) server_negotiation.cc:408] Connection header received
0929 01:56:28.956650 (+ 826us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:56:28.956657 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:56:28.956776 (+ 119us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:56:28.956880 (+ 104us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:56:28.958078 (+ 1198us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:56:28.959939 (+ 1861us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:56:29.088101 (+128162us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:41693: BlockingRecv error: recv got EOF from 127.0.71.65:41693 (error 108)
Metrics: {"server-negotiator.queue_time_us":1528,"thread_start_us":1385,"threads_started":1}
W20260929 01:56:29.089540 32518 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:56:28.930675 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:42293 (local address 127.0.71.126:34931)
0929 01:56:28.947363 (+ 16688us) server_negotiation.cc:207] Beginning negotiation
0929 01:56:28.947371 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0929 01:56:28.951697 (+ 4326us) server_negotiation.cc:408] Connection header received
0929 01:56:28.951966 (+ 269us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:56:28.951976 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:56:28.952147 (+ 171us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:56:28.954790 (+ 2643us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:56:28.957374 (+ 2584us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:56:28.959002 (+ 1628us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:56:29.088141 (+129139us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:42293: BlockingRecv error: recv got EOF from 127.0.71.65:42293 (error 108)
Metrics: {"server-negotiator.queue_time_us":16519,"thread_start_us":16384,"threads_started":1}
I20260929 01:56:29.101202 32180 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:29.224524 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 32117
I20260929 01:56:29.249473 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 31438
I20260929 01:56:29.251123 31500 generic_service.cc:196] Checking for leaks (request via RPC)
W20260929 01:56:29.279301 31528 consensus_peers.cc:597] T 00000000000000000000000000000000 P 7863c2a453d140e5b2093a08c0aa3415 -> Peer 12cdff735da147c59723d8561237671d (127.0.71.126:34931): Couldn't send request to peer 12cdff735da147c59723d8561237671d. Status: Network error: Client connection negotiation failed: client connection to 127.0.71.126:34931: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 01:56:29.374266 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 31438
I20260929 01:56:29.399385 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 31508
I20260929 01:56:29.400888 31572 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:29.531095 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 31508
I20260929 01:56:29.555742 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 31775
I20260929 01:56:29.557482 31946 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:29.680352 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 31775
2026-09-29T01:56:29Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestAddMaster/1 (8562 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/2
2026-09-29T01:56:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:56:29Z Disabled control of system clock
2026-09-29T01:56:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:56:29Z Disabled control of system clock
2026-09-29T01:56:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:56:29Z Disabled control of system clock
2026-09-29T01:56:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:56:29Z Disabled control of system clock
2026-09-29T01:56:29Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:56:29Z Disabled control of system clock
I20260929 01:56:29.959280 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:40625
--webserver_interface=127.0.71.126
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:38211,127.0.71.85:36585,127.0.71.86:37235,127.0.71.87:32931,127.0.71.88:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:40625,127.0.71.125:39489,127.0.71.124:40767 with env {}
W20260929 01:56:30.369153 32541 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:30.369545 32541 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:30.369702 32541 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:30.379180 32541 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:56:30.379339 32541 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:30.379400 32541 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:56:30.379493 32541 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:56:30.391047 32541 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38211,127.0.71.85:36585,127.0.71.86:37235,127.0.71.87:32931,127.0.71.88:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:40625,127.0.71.125:39489,127.0.71.124:40767
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:40625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:30.393293 32541 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:30.395655 32541 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:30.406831 32546 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:30.407330 32547 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:30.407437 32549 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:30.408821 32541 server_base.cc:1061] running on GCE node
I20260929 01:56:30.410482 32541 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:30.412806 32541 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:30.416428 32541 hybrid_clock.cc:648] HybridClock initialized: now 1790646990416219 us; error 114 us; skew 500 ppm
I20260929 01:56:30.417135 32541 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:30.419888 32541 webserver.cc:533] Webserver started at http://127.0.71.126:46807/ using document root <none> and password file <none>
I20260929 01:56:30.421044 32541 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:30.421192 32541 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:30.421624 32541 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:56:30.424585 32541 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "2a9bfa0f293048868a9f9ed43c8bbedd"
format_stamp: "Formatted at 2026-09-29 01:56:30 on dist-test-slave-vz23"
I20260929 01:56:30.425601 32541 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "2a9bfa0f293048868a9f9ed43c8bbedd"
format_stamp: "Formatted at 2026-09-29 01:56:30 on dist-test-slave-vz23"
I20260929 01:56:30.432359 32541 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.004s sys 0.002s
I20260929 01:56:30.436614 32555 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:30.438344 32541 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:56:30.438587 32541 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "2a9bfa0f293048868a9f9ed43c8bbedd"
format_stamp: "Formatted at 2026-09-29 01:56:30 on dist-test-slave-vz23"
I20260929 01:56:30.438855 32541 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:30.485334 32541 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:30.486572 32541 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:30.486974 32541 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:30.513707 32541 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:40625
I20260929 01:56:30.513731 32606 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:40625 every 8 connection(s)
I20260929 01:56:30.515661 32541 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:56:30.521533 32607 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:56:30.525697 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 32541
I20260929 01:56:30.525975 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:56:30.527558 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:39489
--webserver_interface=127.0.71.125
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:38211,127.0.71.85:36585,127.0.71.86:37235,127.0.71.87:32931,127.0.71.88:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:40625,127.0.71.125:39489,127.0.71.124:40767 with env {}
I20260929 01:56:30.531301 32607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:30.554725 32607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:56:30.556111 32556 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.125:39489: connect: Connection refused (error 111)
W20260929 01:56:30.560112 32607 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:39489: Network error: Client connection negotiation failed: client connection to 127.0.71.125:39489: connect: Connection refused (error 111)
I20260929 01:56:30.609789 32607 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } attempt: 1
W20260929 01:56:30.613703 32607 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:39489: Network error: Client connection negotiation failed: client connection to 127.0.71.125:39489: connect: Connection refused (error 111)
I20260929 01:56:30.682101 32607 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } attempt: 2
W20260929 01:56:30.685117 32607 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:39489: Network error: Client connection negotiation failed: client connection to 127.0.71.125:39489: connect: Connection refused (error 111)
I20260929 01:56:30.776577 32607 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } attempt: 3
W20260929 01:56:30.779208 32607 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:39489: Network error: Client connection negotiation failed: client connection to 127.0.71.125:39489: connect: Connection refused (error 111)
I20260929 01:56:30.923897 32607 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } attempt: 4
W20260929 01:56:30.927062 32607 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:39489: Network error: Client connection negotiation failed: client connection to 127.0.71.125:39489: connect: Connection refused (error 111)
W20260929 01:56:30.984390 32611 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:30.985122 32611 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:30.985306 32611 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:31.006153 32611 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:56:31.006335 32611 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:31.006461 32611 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:56:31.006569 32611 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:56:31.032085 32611 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38211,127.0.71.85:36585,127.0.71.86:37235,127.0.71.87:32931,127.0.71.88:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:40625,127.0.71.125:39489,127.0.71.124:40767
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:39489
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:31.034850 32611 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:31.037753 32611 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:31.049535 32618 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:31.049492 32617 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:31.051895 32620 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:31.054241 32611 server_base.cc:1061] running on GCE node
I20260929 01:56:31.055641 32611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:31.058790 32611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:31.062358 32611 hybrid_clock.cc:648] HybridClock initialized: now 1790646991062089 us; error 167 us; skew 500 ppm
I20260929 01:56:31.063108 32611 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:31.067142 32611 webserver.cc:533] Webserver started at http://127.0.71.125:46315/ using document root <none> and password file <none>
I20260929 01:56:31.068648 32611 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:31.068843 32611 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:31.069396 32611 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:56:31.073791 32611 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-1/data/instance:
uuid: "f8484e57b9504f259fc620b645926502"
format_stamp: "Formatted at 2026-09-29 01:56:31 on dist-test-slave-vz23"
I20260929 01:56:31.075309 32611 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-1/wal/instance:
uuid: "f8484e57b9504f259fc620b645926502"
format_stamp: "Formatted at 2026-09-29 01:56:31 on dist-test-slave-vz23"
I20260929 01:56:31.087641 32611 fs_manager.cc:696] Time spent creating directory manager: real 0.012s user 0.012s sys 0.000s
I20260929 01:56:31.094779 32626 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:31.097680 32611 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.005s sys 0.001s
I20260929 01:56:31.097978 32611 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "f8484e57b9504f259fc620b645926502"
format_stamp: "Formatted at 2026-09-29 01:56:31 on dist-test-slave-vz23"
I20260929 01:56:31.098356 32611 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:31.136991 32611 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:31.138712 32611 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:31.139153 32611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:31.171871 32611 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:39489
I20260929 01:56:31.171801 32677 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:39489 every 8 connection(s)
I20260929 01:56:31.175124 32611 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:56:31.180526 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 32611
I20260929 01:56:31.180751 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:56:31.181890 32678 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:56:31.182210 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:40767
--webserver_interface=127.0.71.124
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:38211,127.0.71.85:36585,127.0.71.86:37235,127.0.71.87:32931,127.0.71.88:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:40625,127.0.71.125:39489,127.0.71.124:40767 with env {}
I20260929 01:56:31.192428 32678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:31.223760 32678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:31.226411 32607 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } attempt: 5
I20260929 01:56:31.240091 32607 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:31.240206 32678 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:56:31.242162 32628 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40767: connect: Connection refused (error 111)
W20260929 01:56:31.243198 32607 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:40767: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40767: connect: Connection refused (error 111)
W20260929 01:56:31.244594 32678 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:40767: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40767: connect: Connection refused (error 111)
I20260929 01:56:31.294064 32678 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } attempt: 1
I20260929 01:56:31.294507 32607 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } attempt: 1
W20260929 01:56:31.297253 32607 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:40767: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40767: connect: Connection refused (error 111)
W20260929 01:56:31.297372 32678 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:40767: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40767: connect: Connection refused (error 111)
I20260929 01:56:31.365638 32607 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } attempt: 2
I20260929 01:56:31.365712 32678 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } attempt: 2
W20260929 01:56:31.368961 32678 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:40767: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40767: connect: Connection refused (error 111)
W20260929 01:56:31.369143 32607 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:40767: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40767: connect: Connection refused (error 111)
I20260929 01:56:31.460388 32678 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } attempt: 3
W20260929 01:56:31.463568 32678 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:40767: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40767: connect: Connection refused (error 111)
I20260929 01:56:31.475440 32607 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } attempt: 3
W20260929 01:56:31.478834 32607 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:40767: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40767: connect: Connection refused (error 111)
I20260929 01:56:31.606859 32678 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } attempt: 4
W20260929 01:56:31.609589 32678 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:40767: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40767: connect: Connection refused (error 111)
W20260929 01:56:31.617481 32680 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:31.617900 32680 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:31.617981 32680 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:31.627632 32680 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:56:31.627764 32680 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:31.627849 32680 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:56:31.627897 32680 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:56:31.639393 32680 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38211,127.0.71.85:36585,127.0.71.86:37235,127.0.71.87:32931,127.0.71.88:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:40625,127.0.71.125:39489,127.0.71.124:40767
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:40767
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:31.641443 32680 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:31.643642 32680 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:31.655000 32691 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:31.655390 32690 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:31.655414 32693 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:31.656158 32607 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } attempt: 4
I20260929 01:56:31.656584 32680 server_base.cc:1061] running on GCE node
I20260929 01:56:31.657823 32680 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20260929 01:56:31.658613 32607 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:40767: Network error: Client connection negotiation failed: client connection to 127.0.71.124:40767: connect: Connection refused (error 111)
I20260929 01:56:31.660066 32680 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:31.663617 32680 hybrid_clock.cc:648] HybridClock initialized: now 1790646991663357 us; error 163 us; skew 500 ppm
I20260929 01:56:31.664351 32680 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:31.667491 32680 webserver.cc:533] Webserver started at http://127.0.71.124:43947/ using document root <none> and password file <none>
I20260929 01:56:31.668619 32680 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:31.668792 32680 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:31.669256 32680 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:56:31.672605 32680 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-2/data/instance:
uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b"
format_stamp: "Formatted at 2026-09-29 01:56:31 on dist-test-slave-vz23"
I20260929 01:56:31.673719 32680 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-2/wal/instance:
uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b"
format_stamp: "Formatted at 2026-09-29 01:56:31 on dist-test-slave-vz23"
I20260929 01:56:31.681743 32680 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260929 01:56:31.686655 32699 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:31.688871 32680 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.004s
I20260929 01:56:31.689157 32680 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b"
format_stamp: "Formatted at 2026-09-29 01:56:31 on dist-test-slave-vz23"
I20260929 01:56:31.689508 32680 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:31.708125 32680 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:31.709358 32680 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:31.709726 32680 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:31.737039 32680 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:40767
I20260929 01:56:31.737025 32750 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:40767 every 8 connection(s)
I20260929 01:56:31.739257 32680 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:56:31.745322 32751 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:56:31.749372 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 32680
I20260929 01:56:31.749616 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:56:31.754215 32751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:31.774478 32751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:31.782959 32751 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:31.796948 32751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b: Bootstrap starting.
I20260929 01:56:31.800930 32751 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b: Neither blocks nor log segments found. Creating new log.
I20260929 01:56:31.802479 32751 log.cc:826] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b: Log is configured to *not* fsync() on all Append() calls
I20260929 01:56:31.805986 32751 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b: No bootstrap required, opened a new log
I20260929 01:56:31.812744 32751 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } }
I20260929 01:56:31.813211 32751 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:56:31.813375 32751 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a142b9d6f7b4e7497c0e3d4eb74d73b, State: Initialized, Role: FOLLOWER
I20260929 01:56:31.814211 32751 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } }
I20260929 01:56:31.815411 32760 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } }
I20260929 01:56:31.815711 32760 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:31.816825 32751 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:56:31.831853 303 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:56:31.832156 303 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:56:31.827039 32680 master_runner.cc:187] Error getting master registration for 127.0.71.126:40625: OK, instance_id { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" instance_seqno: 1790646990507052 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:56:31.833989 32680 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:56:31.908907 32678 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } attempt: 5
I20260929 01:56:31.921656 32678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502: Bootstrap starting.
I20260929 01:56:31.927426 32678 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502: Neither blocks nor log segments found. Creating new log.
I20260929 01:56:31.929080 32678 log.cc:826] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502: Log is configured to *not* fsync() on all Append() calls
I20260929 01:56:31.932570 32678 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502: No bootstrap required, opened a new log
I20260929 01:56:31.935935 32607 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } attempt: 5
I20260929 01:56:31.940081 32678 raft_consensus.cc:359] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } }
I20260929 01:56:31.940747 32678 raft_consensus.cc:385] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:56:31.940932 32678 raft_consensus.cc:740] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f8484e57b9504f259fc620b645926502, State: Initialized, Role: FOLLOWER
I20260929 01:56:31.941879 32678 consensus_queue.cc:260] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } }
I20260929 01:56:31.943128 306 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } }
I20260929 01:56:31.943574 306 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:31.945437 32678 sys_catalog.cc:565] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:56:31.953430 32607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd: Bootstrap starting.
I20260929 01:56:31.959466 32607 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd: Neither blocks nor log segments found. Creating new log.
W20260929 01:56:31.962437 32705 tablet.cc:2433] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:56:31.963987 32607 log.cc:826] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd: Log is configured to *not* fsync() on all Append() calls
I20260929 01:56:31.958582 32611 master_runner.cc:187] Error getting master registration for 127.0.71.126:40625: OK, instance_id { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" instance_seqno: 1790646990507052 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:56:31.966472 32611 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:56:31.973816 317 catalog_manager.cc:1607] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:56:31.974004 317 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:56:31.975354 32607 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd: No bootstrap required, opened a new log
I20260929 01:56:31.982134 32607 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } }
I20260929 01:56:31.982571 32607 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:56:31.982756 32607 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a9bfa0f293048868a9f9ed43c8bbedd, State: Initialized, Role: FOLLOWER
I20260929 01:56:31.983609 32607 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } }
I20260929 01:56:31.984897 320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } }
I20260929 01:56:31.985321 320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:31.986353 32607 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:56:31.991181 32561 tablet.cc:2433] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:56:32.016431 331 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:56:32.016786 331 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:56:32.017318 32541 master_runner.cc:264] No leader master found from master 2a9bfa0f293048868a9f9ed43c8bbedd
I20260929 01:56:32.017555 32541 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:56:32.143436 32632 tablet.cc:2433] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:56:32.202346 320 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:56:32.202711 320 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } }
I20260929 01:56:32.205303 320 leader_election.cc:290] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers f8484e57b9504f259fc620b645926502 (127.0.71.125:39489), 2a142b9d6f7b4e7497c0e3d4eb74d73b (127.0.71.124:40767)
I20260929 01:56:32.207436 32652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8484e57b9504f259fc620b645926502" is_pre_election: true
I20260929 01:56:32.208180 32652 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a9bfa0f293048868a9f9ed43c8bbedd in term 0.
I20260929 01:56:32.209338 32726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" is_pre_election: true
I20260929 01:56:32.210091 32726 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 2a9bfa0f293048868a9f9ed43c8bbedd in term 0.
I20260929 01:56:32.211755 32556 leader_election.cc:304] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a9bfa0f293048868a9f9ed43c8bbedd, f8484e57b9504f259fc620b645926502; no voters:
I20260929 01:56:32.213686 320 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:56:32.214380 320 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:56:32.214579 320 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:56:32.224712 320 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } }
I20260929 01:56:32.228937 32652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8484e57b9504f259fc620b645926502"
I20260929 01:56:32.229305 32652 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:56:32.234184 32652 raft_consensus.cc:2468] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a9bfa0f293048868a9f9ed43c8bbedd in term 1.
I20260929 01:56:32.240934 32556 leader_election.cc:304] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2a9bfa0f293048868a9f9ed43c8bbedd, f8484e57b9504f259fc620b645926502; no voters:
I20260929 01:56:32.247334 32726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b"
I20260929 01:56:32.247653 32726 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:56:32.258085 32726 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 2a9bfa0f293048868a9f9ed43c8bbedd in term 1.
I20260929 01:56:32.262279 320 leader_election.cc:290] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [CANDIDATE]: Term 1 election: Requested vote from peers f8484e57b9504f259fc620b645926502 (127.0.71.125:39489), 2a142b9d6f7b4e7497c0e3d4eb74d73b (127.0.71.124:40767)
I20260929 01:56:32.263093 320 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:56:32.263826 320 raft_consensus.cc:697] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [term 1 LEADER]: Becoming Leader. State: Replica: 2a9bfa0f293048868a9f9ed43c8bbedd, State: Running, Role: LEADER
I20260929 01:56:32.264845 320 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } }
I20260929 01:56:32.280371 333 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a9bfa0f293048868a9f9ed43c8bbedd. Latest consensus state: current_term: 1 leader_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } }
I20260929 01:56:32.280853 333 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:32.288421 335 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:56:32.291407 335 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:56:32.319607 32726 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [term 1 FOLLOWER]: Refusing update from remote peer 2a9bfa0f293048868a9f9ed43c8bbedd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:56:32.321472 32652 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [term 1 FOLLOWER]: Refusing update from remote peer 2a9bfa0f293048868a9f9ed43c8bbedd: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:56:32.325981 320 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 01:56:32.331133 320 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:56:32.380904 337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a9bfa0f293048868a9f9ed43c8bbedd. Latest consensus state: current_term: 1 leader_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } }
I20260929 01:56:32.381397 337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:32.387921 336 mvcc.cc:204] Tried to move back new op lower bound from 7334490080521330688 to 7334490080338370560. Current Snapshot: MvccSnapshot[applied={T|T < 7334490080521330688}]
I20260929 01:56:32.389238 306 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a9bfa0f293048868a9f9ed43c8bbedd. Latest consensus state: current_term: 1 leader_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } }
I20260929 01:56:32.389556 306 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:32.395274 338 mvcc.cc:204] Tried to move back new op lower bound from 7334490080521330688 to 7334490080338370560. Current Snapshot: MvccSnapshot[applied={T|T < 7334490080521330688}]
I20260929 01:56:32.396660 337 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } }
I20260929 01:56:32.396989 337 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:32.397229 320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } }
I20260929 01:56:32.397979 339 mvcc.cc:204] Tried to move back new op lower bound from 7334490080521330688 to 7334490080338370560. Current Snapshot: MvccSnapshot[applied={T|T < 7334490080521330688 or (T in {7334490080521330688})}]
I20260929 01:56:32.401968 320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:32.402432 320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } }
I20260929 01:56:32.403126 320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:32.412801 335 catalog_manager.cc:1396] Generated new cluster ID: 08bef3c576944ef8b334861e861744cb
I20260929 01:56:32.412981 335 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:56:32.413204 306 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } }
I20260929 01:56:32.413497 306 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:32.462036 335 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 01:56:32.465241 335 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:56:32.495882 335 catalog_manager.cc:6219] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd: Generated new TSK 0
I20260929 01:56:32.497282 335 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:56:32.503101 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:40625,127.0.71.125:39489,127.0.71.124:40767
--builtin_ntp_servers=127.0.71.84:38211,127.0.71.85:36585,127.0.71.86:37235,127.0.71.87:32931,127.0.71.88:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:56:32.835129 303 catalog_manager.cc:1308] Loaded cluster ID: 08bef3c576944ef8b334861e861744cb
I20260929 01:56:32.835299 303 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b: loading cluster ID for follower catalog manager: success
I20260929 01:56:32.840868 303 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b: acquiring CA information for follower catalog manager: success
I20260929 01:56:32.844179 303 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:56:32.976392 317 catalog_manager.cc:1308] Loaded cluster ID: 08bef3c576944ef8b334861e861744cb
I20260929 01:56:32.976639 317 catalog_manager.cc:1601] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502: loading cluster ID for follower catalog manager: success
W20260929 01:56:32.978480 347 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:32.978835 347 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:32.979015 347 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260929 01:56:32.980222 317 catalog_manager.cc:1623] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502: acquiring CA information for follower catalog manager: success
I20260929 01:56:32.983067 317 catalog_manager.cc:1651] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 01:56:32.989114 347 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:32.989308 347 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:56:33.001904 347 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38211,127.0.71.85:36585,127.0.71.86:37235,127.0.71.87:32931,127.0.71.88:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:40625,127.0.71.125:39489,127.0.71.124:40767
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:33.003911 347 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:33.006366 347 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:33.021373 353 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:33.021531 352 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:33.022517 355 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:33.023576 347 server_base.cc:1061] running on GCE node
I20260929 01:56:33.024303 347 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:33.025779 347 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:33.027127 347 hybrid_clock.cc:648] HybridClock initialized: now 1790646993026979 us; error 110 us; skew 500 ppm
I20260929 01:56:33.027513 347 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:33.030408 347 webserver.cc:533] Webserver started at http://127.0.71.65:37983/ using document root <none> and password file <none>
I20260929 01:56:33.031371 347 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:33.031522 347 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:33.031997 347 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:56:33.034858 347 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "075b7745e5904472960152046484ddf5"
format_stamp: "Formatted at 2026-09-29 01:56:33 on dist-test-slave-vz23"
I20260929 01:56:33.035707 347 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "075b7745e5904472960152046484ddf5"
format_stamp: "Formatted at 2026-09-29 01:56:33 on dist-test-slave-vz23"
I20260929 01:56:33.042717 347 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260929 01:56:33.048393 362 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:33.051091 347 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260929 01:56:33.051441 347 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "075b7745e5904472960152046484ddf5"
format_stamp: "Formatted at 2026-09-29 01:56:33 on dist-test-slave-vz23"
I20260929 01:56:33.051743 347 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:33.089345 347 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:33.090950 347 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:33.091333 347 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:33.092900 347 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:56:33.096349 347 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:56:33.096807 347 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 01:56:33.097067 347 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:56:33.097139 347 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:33.144630 347 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:39647
I20260929 01:56:33.144680 474 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:39647 every 8 connection(s)
I20260929 01:56:33.146530 347 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:56:33.151849 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 347
I20260929 01:56:33.152137 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
I20260929 01:56:33.181736 477 heartbeater.cc:344] Connected to a master server at 127.0.71.125:39489
I20260929 01:56:33.181842 475 heartbeater.cc:344] Connected to a master server at 127.0.71.124:40767
I20260929 01:56:33.182258 475 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:33.182264 477 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:33.183454 475 heartbeater.cc:507] Master 127.0.71.124:40767 requested a full tablet report, sending...
I20260929 01:56:33.183481 477 heartbeater.cc:507] Master 127.0.71.125:39489 requested a full tablet report, sending...
I20260929 01:56:33.181823 476 heartbeater.cc:344] Connected to a master server at 127.0.71.126:40625
I20260929 01:56:33.183833 476 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:33.184620 476 heartbeater.cc:507] Master 127.0.71.126:40625 requested a full tablet report, sending...
I20260929 01:56:33.186611 32643 ts_manager.cc:194] Registered new tserver with Master: 075b7745e5904472960152046484ddf5 (127.0.71.65:39647)
I20260929 01:56:33.186654 32716 ts_manager.cc:194] Registered new tserver with Master: 075b7745e5904472960152046484ddf5 (127.0.71.65:39647)
I20260929 01:56:33.189035 32572 ts_manager.cc:194] Registered new tserver with Master: 075b7745e5904472960152046484ddf5 (127.0.71.65:39647)
I20260929 01:56:33.191165 32572 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:32793
I20260929 01:56:33.192649 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:56:33.195047 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.123:43997
--webserver_interface=127.0.71.123
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:38211,127.0.71.85:36585,127.0.71.86:37235,127.0.71.87:32931,127.0.71.88:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:40625,127.0.71.125:39489,127.0.71.124:40767,127.0.71.123:43997 with env {}
W20260929 01:56:33.621228 485 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:33.622103 485 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:33.622264 485 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:33.634670 485 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:56:33.634835 485 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:33.634900 485 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:56:33.634948 485 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:56:33.648841 485 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38211,127.0.71.85:36585,127.0.71.86:37235,127.0.71.87:32931,127.0.71.88:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:40625,127.0.71.125:39489,127.0.71.124:40767,127.0.71.123:43997
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.123:43997
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.0.71.123
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:33.650913 485 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:33.653632 485 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:33.665920 490 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:33.666887 491 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:33.666887 493 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:33.669340 485 server_base.cc:1061] running on GCE node
I20260929 01:56:33.670984 485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:33.673693 485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:33.677291 485 hybrid_clock.cc:648] HybridClock initialized: now 1790646993677010 us; error 175 us; skew 500 ppm
I20260929 01:56:33.678231 485 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:33.682879 485 webserver.cc:533] Webserver started at http://127.0.71.123:43033/ using document root <none> and password file <none>
I20260929 01:56:33.684801 485 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:33.685078 485 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:33.685571 485 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:56:33.689239 485 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-3/data/instance:
uuid: "23a54f0804fc43e0ad31046a3b20d7b5"
format_stamp: "Formatted at 2026-09-29 01:56:33 on dist-test-slave-vz23"
I20260929 01:56:33.690743 485 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-3/wal/instance:
uuid: "23a54f0804fc43e0ad31046a3b20d7b5"
format_stamp: "Formatted at 2026-09-29 01:56:33 on dist-test-slave-vz23"
I20260929 01:56:33.698830 485 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.000s sys 0.008s
I20260929 01:56:33.704362 499 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:33.706755 485 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260929 01:56:33.707068 485 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "23a54f0804fc43e0ad31046a3b20d7b5"
format_stamp: "Formatted at 2026-09-29 01:56:33 on dist-test-slave-vz23"
I20260929 01:56:33.707376 485 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:33.744360 485 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:33.746076 485 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:33.746510 485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:33.773038 485 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.123:43997
I20260929 01:56:33.773058 550 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.123:43997 every 8 connection(s)
I20260929 01:56:33.775084 485 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
I20260929 01:56:33.781054 551 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:56:33.781546 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 485
I20260929 01:56:33.790961 551 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:33.805030 551 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:33.815689 551 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:33.825605 551 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.123" port: 43997 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:33.841820 551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5: Bootstrap starting.
I20260929 01:56:33.847239 551 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5: Neither blocks nor log segments found. Creating new log.
I20260929 01:56:33.849365 551 log.cc:826] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5: Log is configured to *not* fsync() on all Append() calls
I20260929 01:56:33.853747 551 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5: No bootstrap required, opened a new log
I20260929 01:56:33.860319 551 raft_consensus.cc:359] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 43997 } }
I20260929 01:56:33.860843 551 raft_consensus.cc:385] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:56:33.860975 551 raft_consensus.cc:740] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 23a54f0804fc43e0ad31046a3b20d7b5, State: Initialized, Role: FOLLOWER
I20260929 01:56:33.862478 551 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 43997 } }
I20260929 01:56:33.863226 560 raft_consensus.cc:493] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:56:33.863859 560 raft_consensus.cc:515] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 43997 } }
I20260929 01:56:33.865958 551 sys_catalog.cc:565] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:56:33.867381 561 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 43997 } } }
I20260929 01:56:33.868055 561 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:33.869971 32652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "f8484e57b9504f259fc620b645926502" is_pre_election: true
I20260929 01:56:33.870134 32582 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" is_pre_election: true
I20260929 01:56:33.870406 560 leader_election.cc:290] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2a9bfa0f293048868a9f9ed43c8bbedd (127.0.71.126:40625), f8484e57b9504f259fc620b645926502 (127.0.71.125:39489), 2a142b9d6f7b4e7497c0e3d4eb74d73b (127.0.71.124:40767)
I20260929 01:56:33.870838 32726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" is_pre_election: true
I20260929 01:56:33.871914 501 leader_election.cc:304] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 23a54f0804fc43e0ad31046a3b20d7b5; no voters: 2a142b9d6f7b4e7497c0e3d4eb74d73b, 2a9bfa0f293048868a9f9ed43c8bbedd
I20260929 01:56:33.872679 560 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260929 01:56:33.884524 572 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:56:33.884737 572 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:56:33.896319 485 master_runner.cc:429] Detected that this master 23a54f0804fc43e0ad31046a3b20d7b5 is joining an existing cluster
I20260929 01:56:33.896503 485 catalog_manager.cc:1037] Shutting down the Catalog Manager
I20260929 01:56:33.897696 485 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260929 01:56:33.898234 485 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260929 01:56:33.898499 485 tablet_replica.cc:333] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5: stopping tablet replica
I20260929 01:56:33.900593 485 master_runner.cc:434] Initiating AddMaster RPC to add 127.0.71.123:43997
I20260929 01:56:33.948969 32572 catalog_manager.cc:7351] Initiating ChangeConfig request to add master 127.0.71.123:43997: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5"
member_type: NON_VOTER
last_known_addr {
host: "127.0.71.123"
port: 43997
}
attrs {
promote: true
}
}
dest_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd"
cas_config_opid_index: -1
I20260929 01:56:33.951133 32572 consensus_queue.cc:237] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } }
I20260929 01:56:33.955650 32652 raft_consensus.cc:1275] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [term 1 FOLLOWER]: Refusing update from remote peer 2a9bfa0f293048868a9f9ed43c8bbedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 01:56:33.956005 32726 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [term 1 FOLLOWER]: Refusing update from remote peer 2a9bfa0f293048868a9f9ed43c8bbedd: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20260929 01:56:33.956840 32556 consensus_peers.cc:597] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd -> Peer 23a54f0804fc43e0ad31046a3b20d7b5 (127.0.71.123:43997): Couldn't send request to peer 23a54f0804fc43e0ad31046a3b20d7b5. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260929 01:56:33.957031 360 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:56:33.957436 583 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [LEADER]: Connected to new peer: Peer: permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:56:33.962247 360 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 23a54f0804fc43e0ad31046a3b20d7b5 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } } }
I20260929 01:56:33.963526 32726 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 23a54f0804fc43e0ad31046a3b20d7b5 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } } }
I20260929 01:56:33.965592 32652 raft_consensus.cc:2955] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 23a54f0804fc43e0ad31046a3b20d7b5 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } } }
I20260929 01:56:33.967922 360 catalog_manager.cc:7355] Successfully completed master ChangeConfig request to add master 127.0.71.123:43997
I20260929 01:56:33.969067 485 master.cc:562] Master@127.0.71.123:43997 shutting down...
I20260929 01:56:33.969126 583 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } } }
I20260929 01:56:33.969513 583 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:33.970839 360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } } }
I20260929 01:56:33.971154 360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:33.971442 360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } } }
I20260929 01:56:33.971642 360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:33.981405 589 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } } }
I20260929 01:56:33.981801 589 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:33.980257 588 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } } }
I20260929 01:56:33.982316 588 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:34.004366 485 master.cc:584] Master@127.0.71.123:43997 shutdown complete.
I20260929 01:56:34.004703 485 master_runner.cc:306] Clearing existing system tablet
I20260929 01:56:34.005167 485 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:34.005340 485 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:34.009716 485 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.001s
I20260929 01:56:34.011859 591 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:34.012426 485 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:34.012811 485 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 01:56:34.012955 485 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "23a54f0804fc43e0ad31046a3b20d7b5"
format_stamp: "Formatted at 2026-09-29 01:56:33 on dist-test-slave-vz23"
I20260929 01:56:34.017444 485 ts_tablet_manager.cc:1962] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:56:34.023909 485 ts_tablet_manager.cc:1975] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 01:56:34.024106 485 log.cc:1199] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260929 01:56:34.024724 485 ts_tablet_manager.cc:1996] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5: Deleting consensus metadata
I20260929 01:56:34.025029 485 master_runner.cc:316] Copying system tablet from 127.0.71.126:40625
I20260929 01:56:34.029124 485 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5: tablet copy: Beginning tablet copy session from remote peer at address 127.0.71.126:40625
I20260929 01:56:34.039976 32592 tablet_copy_service.cc:140] P 2a9bfa0f293048868a9f9ed43c8bbedd: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 23a54f0804fc43e0ad31046a3b20d7b5 ({username='slave'} at 127.0.0.1:58120)
I20260929 01:56:34.040371 32592 tablet_copy_service.cc:161] P 2a9bfa0f293048868a9f9ed43c8bbedd: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 23a54f0804fc43e0ad31046a3b20d7b5 at {username='slave'} at 127.0.0.1:58120: session id = 23a54f0804fc43e0ad31046a3b20d7b5-00000000000000000000000000000000
I20260929 01:56:34.045358 32592 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 01:56:34.049084 485 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 01:56:34.058435 485 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5: tablet copy: Starting download of 0 data blocks...
I20260929 01:56:34.058841 485 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5: tablet copy: Starting download of 1 WAL segments...
I20260929 01:56:34.063113 485 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 01:56:34.069329 32592 tablet_copy_service.cc:342] P 2a9bfa0f293048868a9f9ed43c8bbedd: Request end of tablet copy session 23a54f0804fc43e0ad31046a3b20d7b5-00000000000000000000000000000000 received from {username='slave'} at 127.0.0.1:58120
I20260929 01:56:34.069629 32592 tablet_copy_service.cc:434] P 2a9bfa0f293048868a9f9ed43c8bbedd: ending tablet copy session 23a54f0804fc43e0ad31046a3b20d7b5-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 23a54f0804fc43e0ad31046a3b20d7b5
W20260929 01:56:34.078403 485 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260929 01:56:34.083757 485 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:34.088842 598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:34.089156 599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:34.090710 601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:34.091347 485 server_base.cc:1061] running on GCE node
I20260929 01:56:34.091740 485 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:34.092828 485 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:34.094215 485 hybrid_clock.cc:648] HybridClock initialized: now 1790646994094025 us; error 92 us; skew 500 ppm
I20260929 01:56:34.094590 485 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:34.096402 485 webserver.cc:533] Webserver started at http://127.0.71.123:32839/ using document root <none> and password file <none>
I20260929 01:56:34.096870 485 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:34.096980 485 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:34.099660 485 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260929 01:56:34.101694 607 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:34.102317 485 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:56:34.102478 485 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "23a54f0804fc43e0ad31046a3b20d7b5"
format_stamp: "Formatted at 2026-09-29 01:56:33 on dist-test-slave-vz23"
I20260929 01:56:34.102677 485 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:34.164338 485 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:34.165392 485 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:34.194401 476 heartbeater.cc:499] Master 127.0.71.126:40625 was elected leader, sending a full tablet report...
I20260929 01:56:34.198424 485 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.123:43997
I20260929 01:56:34.198475 658 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.123:43997 every 8 connection(s)
I20260929 01:56:34.199777 485 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
I20260929 01:56:34.206008 659 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:56:34.210291 659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5: Bootstrap starting.
I20260929 01:56:34.232988 659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:56:34.233691 659 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5: Bootstrap complete.
I20260929 01:56:34.235042 659 raft_consensus.cc:359] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } }
I20260929 01:56:34.235347 659 raft_consensus.cc:740] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 23a54f0804fc43e0ad31046a3b20d7b5, State: Initialized, Role: LEARNER
I20260929 01:56:34.235620 659 consensus_queue.cc:260] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } }
I20260929 01:56:34.237228 659 sys_catalog.cc:565] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:56:34.237082 663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } } }
I20260929 01:56:34.237860 663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [sys.catalog]: This master's current role is: LEARNER
I20260929 01:56:34.248135 674 catalog_manager.cc:1308] Loaded cluster ID: 08bef3c576944ef8b334861e861744cb
I20260929 01:56:34.248281 674 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5: loading cluster ID for follower catalog manager: success
I20260929 01:56:34.254081 674 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5: acquiring CA information for follower catalog manager: success
I20260929 01:56:34.257233 674 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:56:34.276578 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 347
I20260929 01:56:34.287645 470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:34.392068 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 347
I20260929 01:56:34.420629 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:39647
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:40625,127.0.71.125:39489,127.0.71.124:40767
--webserver_port=37983
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:38211,127.0.71.85:36585,127.0.71.86:37235,127.0.71.87:32931,127.0.71.88:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:40625,127.0.71.125:39489,127.0.71.124:40767,127.0.71.123:43997 with env {}
I20260929 01:56:34.556315 634 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260929 01:56:34.557195 663 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 2a9bfa0f293048868a9f9ed43c8bbedd. Latest consensus state: current_term: 1 leader_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } } }
I20260929 01:56:34.557500 663 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [sys.catalog]: This master's current role is: LEARNER
W20260929 01:56:34.834517 677 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:34.834932 677 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:34.835162 677 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:34.845036 677 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:34.845294 677 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:56:34.856828 677 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38211,127.0.71.85:36585,127.0.71.86:37235,127.0.71.87:32931,127.0.71.88:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:39647
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=37983
--tserver_master_addrs=127.0.71.126:40625,127.0.71.125:39489,127.0.71.124:40767,127.0.71.123:43997
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:34.858906 677 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:34.861287 677 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:34.874119 684 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:34.874122 686 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:34.874153 683 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:34.875402 677 server_base.cc:1061] running on GCE node
I20260929 01:56:34.876207 677 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:34.877738 677 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:34.881223 677 hybrid_clock.cc:648] HybridClock initialized: now 1790646994880973 us; error 149 us; skew 500 ppm
I20260929 01:56:34.881747 677 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:34.885077 677 webserver.cc:533] Webserver started at http://127.0.71.65:37983/ using document root <none> and password file <none>
I20260929 01:56:34.886183 677 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:34.886399 677 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:34.894224 677 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 01:56:34.898454 692 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:34.900791 677 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:56:34.901057 677 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "075b7745e5904472960152046484ddf5"
format_stamp: "Formatted at 2026-09-29 01:56:33 on dist-test-slave-vz23"
I20260929 01:56:34.902038 677 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:34.943164 677 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:34.944278 677 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:34.944779 677 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:34.946108 677 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:56:34.948678 677 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:56:34.948832 677 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:34.949012 677 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:56:34.949087 677 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:34.993287 677 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:39647
I20260929 01:56:34.993407 804 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:39647 every 8 connection(s)
I20260929 01:56:34.995302 677 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:56:34.996749 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 677
I20260929 01:56:35.036262 806 heartbeater.cc:344] Connected to a master server at 127.0.71.124:40767
I20260929 01:56:35.036971 806 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:35.038647 806 heartbeater.cc:507] Master 127.0.71.124:40767 requested a full tablet report, sending...
I20260929 01:56:35.039431 807 heartbeater.cc:344] Connected to a master server at 127.0.71.126:40625
I20260929 01:56:35.039757 807 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:35.040927 807 heartbeater.cc:507] Master 127.0.71.126:40625 requested a full tablet report, sending...
I20260929 01:56:35.041610 32716 ts_manager.cc:194] Re-registered known tserver with Master: 075b7745e5904472960152046484ddf5 (127.0.71.65:39647)
I20260929 01:56:35.043633 32572 ts_manager.cc:194] Re-registered known tserver with Master: 075b7745e5904472960152046484ddf5 (127.0.71.65:39647)
I20260929 01:56:35.044960 32572 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:50931
I20260929 01:56:35.051508 805 heartbeater.cc:344] Connected to a master server at 127.0.71.123:43997
I20260929 01:56:35.051715 805 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:35.052277 805 heartbeater.cc:507] Master 127.0.71.123:43997 requested a full tablet report, sending...
I20260929 01:56:35.054623 808 heartbeater.cc:344] Connected to a master server at 127.0.71.125:39489
I20260929 01:56:35.054884 808 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:35.055006 624 ts_manager.cc:194] Registered new tserver with Master: 075b7745e5904472960152046484ddf5 (127.0.71.65:39647)
I20260929 01:56:35.055294 808 heartbeater.cc:507] Master 127.0.71.125:39489 requested a full tablet report, sending...
I20260929 01:56:35.057294 32643 ts_manager.cc:194] Re-registered known tserver with Master: 075b7745e5904472960152046484ddf5 (127.0.71.65:39647)
I20260929 01:56:35.062384 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:56:35.077695 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:56:35.077955 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 32541
I20260929 01:56:35.079898 32602 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:35.213980 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 32541
I20260929 01:56:35.213904 360 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } } }
I20260929 01:56:35.214279 360 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [sys.catalog]: This master's current role is: LEADER
W20260929 01:56:35.240355 609 connection.cc:573] server connection from 127.0.0.1:50442 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:56:35.241484 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:40625
--webserver_interface=127.0.71.126
--webserver_port=46807
--builtin_ntp_servers=127.0.71.84:38211,127.0.71.85:36585,127.0.71.86:37235,127.0.71.87:32931,127.0.71.88:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:40625,127.0.71.125:39489,127.0.71.124:40767
--master_addresses=127.0.71.126:40625,127.0.71.125:39489,127.0.71.124:40767,127.0.71.123:43997 with env {}
W20260929 01:56:35.670426 820 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:35.670892 820 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:35.671082 820 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:35.682230 820 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:56:35.682365 820 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:35.682467 820 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:56:35.682550 820 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:56:35.695250 820 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38211,127.0.71.85:36585,127.0.71.86:37235,127.0.71.87:32931,127.0.71.88:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:40625,127.0.71.125:39489,127.0.71.124:40767,127.0.71.123:43997
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:40625
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=46807
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:35.697499 820 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:35.699717 820 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:35.711136 825 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:35.711243 826 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:35.712163 828 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:35.713229 820 server_base.cc:1061] running on GCE node
I20260929 01:56:35.714293 820 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:35.716655 820 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:35.720183 820 hybrid_clock.cc:648] HybridClock initialized: now 1790646995719963 us; error 127 us; skew 500 ppm
I20260929 01:56:35.720870 820 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:35.724097 820 webserver.cc:533] Webserver started at http://127.0.71.126:46807/ using document root <none> and password file <none>
I20260929 01:56:35.725245 820 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:35.725451 820 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:35.733060 820 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.003s
I20260929 01:56:35.737375 834 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:35.740082 820 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260929 01:56:35.740383 820 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "2a9bfa0f293048868a9f9ed43c8bbedd"
format_stamp: "Formatted at 2026-09-29 01:56:30 on dist-test-slave-vz23"
I20260929 01:56:35.741696 820 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:35.785373 820 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:35.786520 820 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:35.786896 820 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:35.814307 820 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:40625
I20260929 01:56:35.814332 885 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:40625 every 8 connection(s)
I20260929 01:56:35.816660 820 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:56:35.819312 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 820
I20260929 01:56:35.819687 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 677
I20260929 01:56:35.827069 886 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:56:35.831257 800 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:35.832495 886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd: Bootstrap starting.
I20260929 01:56:35.861392 886 log.cc:826] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd: Log is configured to *not* fsync() on all Append() calls
I20260929 01:56:35.875000 886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:56:35.875739 886 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd: Bootstrap complete.
I20260929 01:56:35.884667 886 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } }
I20260929 01:56:35.885857 886 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2a9bfa0f293048868a9f9ed43c8bbedd, State: Initialized, Role: FOLLOWER
I20260929 01:56:35.886979 886 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } }
I20260929 01:56:35.888298 892 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } } }
I20260929 01:56:35.888761 892 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:35.889712 886 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:56:35.904770 903 catalog_manager.cc:1308] Loaded cluster ID: 08bef3c576944ef8b334861e861744cb
I20260929 01:56:35.905068 903 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd: loading cluster ID for follower catalog manager: success
I20260929 01:56:35.914634 903 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd: acquiring CA information for follower catalog manager: success
I20260929 01:56:35.919190 903 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:56:35.955361 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 677
I20260929 01:56:35.974814 820 master_runner.cc:264] No leader master found from master 2a9bfa0f293048868a9f9ed43c8bbedd
I20260929 01:56:35.975075 820 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:56:35.978991 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:39647
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:40625,127.0.71.125:39489,127.0.71.124:40767
--webserver_port=37983
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:38211,127.0.71.85:36585,127.0.71.86:37235,127.0.71.87:32931,127.0.71.88:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:40625,127.0.71.125:39489,127.0.71.124:40767,127.0.71.123:43997 with env {}
I20260929 01:56:36.424232 911 raft_consensus.cc:493] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2a9bfa0f293048868a9f9ed43c8bbedd)
I20260929 01:56:36.424569 911 raft_consensus.cc:515] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } }
I20260929 01:56:36.426332 911 leader_election.cc:290] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a9bfa0f293048868a9f9ed43c8bbedd (127.0.71.126:40625), 2a142b9d6f7b4e7497c0e3d4eb74d73b (127.0.71.124:40767)
I20260929 01:56:36.427285 32726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f8484e57b9504f259fc620b645926502" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" is_pre_election: true
I20260929 01:56:36.436163 861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f8484e57b9504f259fc620b645926502" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" is_pre_election: true
I20260929 01:56:36.436787 861 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate f8484e57b9504f259fc620b645926502 in term 1.
I20260929 01:56:36.437552 32627 leader_election.cc:304] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a9bfa0f293048868a9f9ed43c8bbedd, f8484e57b9504f259fc620b645926502; no voters: 2a142b9d6f7b4e7497c0e3d4eb74d73b
I20260929 01:56:36.438056 911 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 01:56:36.438249 911 raft_consensus.cc:493] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 2a9bfa0f293048868a9f9ed43c8bbedd)
I20260929 01:56:36.438378 911 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:56:36.442945 911 raft_consensus.cc:515] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } }
I20260929 01:56:36.444115 911 leader_election.cc:290] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [CANDIDATE]: Term 2 election: Requested vote from peers 2a9bfa0f293048868a9f9ed43c8bbedd (127.0.71.126:40625), 2a142b9d6f7b4e7497c0e3d4eb74d73b (127.0.71.124:40767)
I20260929 01:56:36.444922 32726 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f8484e57b9504f259fc620b645926502" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b"
I20260929 01:56:36.444962 861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "f8484e57b9504f259fc620b645926502" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd"
I20260929 01:56:36.445263 861 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:56:36.452500 861 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate f8484e57b9504f259fc620b645926502 in term 2.
I20260929 01:56:36.453409 32627 leader_election.cc:304] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 2a9bfa0f293048868a9f9ed43c8bbedd, f8484e57b9504f259fc620b645926502; no voters: 2a142b9d6f7b4e7497c0e3d4eb74d73b
I20260929 01:56:36.453899 911 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:56:36.454509 911 raft_consensus.cc:697] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [term 2 LEADER]: Becoming Leader. State: Replica: f8484e57b9504f259fc620b645926502, State: Running, Role: LEADER
I20260929 01:56:36.455045 911 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } }
W20260929 01:56:36.457099 910 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:36.457657 910 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:36.457814 910 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260929 01:56:36.457828 916 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8484e57b9504f259fc620b645926502. Latest consensus state: current_term: 2 leader_uuid: "f8484e57b9504f259fc620b645926502" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } } }
I20260929 01:56:36.458178 916 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:36.459076 918 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:56:36.461249 918 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:56:36.462514 918 catalog_manager.cc:1308] Loaded cluster ID: 08bef3c576944ef8b334861e861744cb
I20260929 01:56:36.462618 918 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:56:36.463964 918 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:56:36.465006 918 catalog_manager.cc:6230] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502: Loaded TSK: 0
I20260929 01:56:36.466192 918 catalog_manager.cc:1563] Initializing in-progress tserver states...
W20260929 01:56:36.468357 910 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:36.468641 910 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:56:36.481966 910 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38211,127.0.71.85:36585,127.0.71.86:37235,127.0.71.87:32931,127.0.71.88:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:39647
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=37983
--tserver_master_addrs=127.0.71.126:40625,127.0.71.125:39489,127.0.71.124:40767,127.0.71.123:43997
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:36.483945 910 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:36.486310 910 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260929 01:56:36.488979 919 raft_consensus.cc:493] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 2a9bfa0f293048868a9f9ed43c8bbedd)
I20260929 01:56:36.489410 919 raft_consensus.cc:515] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } }
I20260929 01:56:36.490928 919 leader_election.cc:290] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 2a9bfa0f293048868a9f9ed43c8bbedd (127.0.71.126:40625), f8484e57b9504f259fc620b645926502 (127.0.71.125:39489)
I20260929 01:56:36.492547 32652 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "f8484e57b9504f259fc620b645926502" is_pre_election: true
I20260929 01:56:36.500109 861 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" is_pre_election: true
I20260929 01:56:36.501173 861 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 2a142b9d6f7b4e7497c0e3d4eb74d73b in current term 2: Already voted for candidate f8484e57b9504f259fc620b645926502 in this term.
W20260929 01:56:36.502046 923 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:36.503135 32700 leader_election.cc:304] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 2a142b9d6f7b4e7497c0e3d4eb74d73b; no voters: 2a9bfa0f293048868a9f9ed43c8bbedd, f8484e57b9504f259fc620b645926502
W20260929 01:56:36.503888 922 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:36.504529 919 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [term 1 FOLLOWER]: Advancing to term 2
W20260929 01:56:36.505559 925 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:36.508185 910 server_base.cc:1061] running on GCE node
I20260929 01:56:36.508662 919 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:56:36.509346 910 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:36.510866 910 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:36.514344 910 hybrid_clock.cc:648] HybridClock initialized: now 1790646996514127 us; error 139 us; skew 500 ppm
I20260929 01:56:36.514878 910 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:36.517889 910 webserver.cc:533] Webserver started at http://127.0.71.65:37983/ using document root <none> and password file <none>
I20260929 01:56:36.518899 910 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:36.519064 910 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:36.526533 910 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 01:56:36.530822 931 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:36.533030 910 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260929 01:56:36.533324 910 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "075b7745e5904472960152046484ddf5"
format_stamp: "Formatted at 2026-09-29 01:56:33 on dist-test-slave-vz23"
I20260929 01:56:36.534278 910 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:36.555665 910 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:36.556806 910 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:36.557183 910 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:36.558565 910 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:56:36.561175 910 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:56:36.561336 910 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:36.561540 910 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:56:36.561635 910 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:36.603997 910 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:39647
I20260929 01:56:36.604034 1043 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:39647 every 8 connection(s)
I20260929 01:56:36.605985 910 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:56:36.610212 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 910
I20260929 01:56:36.610541 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 910
I20260929 01:56:36.662307 1044 heartbeater.cc:344] Connected to a master server at 127.0.71.123:43997
I20260929 01:56:36.662364 1047 heartbeater.cc:344] Connected to a master server at 127.0.71.125:39489
I20260929 01:56:36.662968 1045 heartbeater.cc:344] Connected to a master server at 127.0.71.124:40767
I20260929 01:56:36.663112 1047 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:36.663192 1044 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:36.663362 1039 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:36.663616 1045 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:36.772821 1046 heartbeater.cc:344] Connected to a master server at 127.0.71.126:40625
I20260929 01:56:36.774204 1046 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:36.775329 1046 heartbeater.cc:507] Master 127.0.71.126:40625 requested a full tablet report, sending...
I20260929 01:56:36.775399 1044 heartbeater.cc:507] Master 127.0.71.123:43997 requested a full tablet report, sending...
I20260929 01:56:36.776088 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 910
I20260929 01:56:36.804829 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:39647
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:40625,127.0.71.125:39489,127.0.71.124:40767
--webserver_port=37983
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:38211,127.0.71.85:36585,127.0.71.86:37235,127.0.71.87:32931,127.0.71.88:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:40625,127.0.71.125:39489,127.0.71.124:40767,127.0.71.123:43997
--fault_before_start=1.0 with env {}
I20260929 01:56:36.952493 634 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [term 1 LEARNER]: Advancing to term 2
I20260929 01:56:36.956615 634 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [term 2 LEARNER]: Refusing update from remote peer f8484e57b9504f259fc620b645926502: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20260929 01:56:36.958259 916 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.001s
I20260929 01:56:36.963424 1061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8484e57b9504f259fc620b645926502. Latest consensus state: current_term: 2 leader_uuid: "f8484e57b9504f259fc620b645926502" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } } }
I20260929 01:56:36.963902 1061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [sys.catalog]: This master's current role is: LEARNER
I20260929 01:56:36.965796 916 raft_consensus.cc:1064] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502: attempting to promote NON_VOTER 23a54f0804fc43e0ad31046a3b20d7b5 to VOTER
W20260929 01:56:36.966228 916 raft_consensus.cc:1068] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502: Unable to promote non-voter 23a54f0804fc43e0ad31046a3b20d7b5: Illegal state: Leader has not yet committed an operation in its own term
I20260929 01:56:37.004348 861 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [term 2 FOLLOWER]: Refusing update from remote peer f8484e57b9504f259fc620b645926502: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20260929 01:56:37.005868 916 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:56:37.011371 1064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8484e57b9504f259fc620b645926502. Latest consensus state: current_term: 2 leader_uuid: "f8484e57b9504f259fc620b645926502" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } } }
I20260929 01:56:37.011935 1064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:37.014847 916 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "f8484e57b9504f259fc620b645926502" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } } }
I20260929 01:56:37.014763 1063 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "f8484e57b9504f259fc620b645926502" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } } }
I20260929 01:56:37.015141 916 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:37.015175 1063 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:37.016726 32726 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [term 2 FOLLOWER]: Refusing update from remote peer f8484e57b9504f259fc620b645926502: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20260929 01:56:37.017740 916 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:56:37.018200 1061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "f8484e57b9504f259fc620b645926502" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } } }
I20260929 01:56:37.018450 1061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [sys.catalog]: This master's current role is: LEARNER
I20260929 01:56:37.020901 1063 raft_consensus.cc:1064] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502: attempting to promote NON_VOTER 23a54f0804fc43e0ad31046a3b20d7b5 to VOTER
I20260929 01:56:37.023222 1063 consensus_queue.cc:237] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: false } }
I20260929 01:56:37.026223 1064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "f8484e57b9504f259fc620b645926502" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } } }
I20260929 01:56:37.026571 1064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:37.029111 634 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [term 2 LEARNER]: Refusing update from remote peer f8484e57b9504f259fc620b645926502: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20260929 01:56:37.030548 916 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 01:56:37.036885 861 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [term 2 FOLLOWER]: Refusing update from remote peer f8484e57b9504f259fc620b645926502: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20260929 01:56:37.038211 1073 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 01:56:37.038558 1071 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [sys.catalog]: SysCatalogTable state changed. Reason: New leader f8484e57b9504f259fc620b645926502. Latest consensus state: current_term: 2 leader_uuid: "f8484e57b9504f259fc620b645926502" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } } }
I20260929 01:56:37.038882 1071 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:37.038625 1075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "f8484e57b9504f259fc620b645926502" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: true } } }
I20260929 01:56:37.039469 32725 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [term 2 FOLLOWER]: Refusing update from remote peer f8484e57b9504f259fc620b645926502: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20260929 01:56:37.039688 1075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:37.041141 1073 consensus_queue.cc:1048] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [LEADER]: Connected to new peer: Peer: permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 01:56:37.041967 916 raft_consensus.cc:2955] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 23a54f0804fc43e0ad31046a3b20d7b5 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: false } } }
I20260929 01:56:37.043566 861 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 23a54f0804fc43e0ad31046a3b20d7b5 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: false } } }
I20260929 01:56:37.047219 1067 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 2 leader_uuid: "f8484e57b9504f259fc620b645926502" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: false } } }
I20260929 01:56:37.047542 1067 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:37.047511 916 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "f8484e57b9504f259fc620b645926502" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: false } } }
I20260929 01:56:37.047757 916 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:37.047730 32726 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 23a54f0804fc43e0ad31046a3b20d7b5 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: false } } }
I20260929 01:56:37.048485 634 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 23a54f0804fc43e0ad31046a3b20d7b5 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: false } } }
I20260929 01:56:37.051319 1064 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "f8484e57b9504f259fc620b645926502" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: false } } }
I20260929 01:56:37.051585 1064 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a9bfa0f293048868a9f9ed43c8bbedd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:37.052855 1075 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "f8484e57b9504f259fc620b645926502" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: false } } }
I20260929 01:56:37.053261 1075 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2a142b9d6f7b4e7497c0e3d4eb74d73b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:37.053572 1061 sys_catalog.cc:455] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "f8484e57b9504f259fc620b645926502" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: false } } }
I20260929 01:56:37.053845 1061 sys_catalog.cc:458] T 00000000000000000000000000000000 P 23a54f0804fc43e0ad31046a3b20d7b5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:56:37.054221 1063 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "f8484e57b9504f259fc620b645926502" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: false } } }
I20260929 01:56:37.054283 1067 sys_catalog.cc:455] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "f8484e57b9504f259fc620b645926502" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "2a9bfa0f293048868a9f9ed43c8bbedd" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 40625 } } peers { permanent_uuid: "f8484e57b9504f259fc620b645926502" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39489 } } peers { permanent_uuid: "2a142b9d6f7b4e7497c0e3d4eb74d73b" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 40767 } } peers { permanent_uuid: "23a54f0804fc43e0ad31046a3b20d7b5" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 43997 } attrs { promote: false } } }
I20260929 01:56:37.054481 1063 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [sys.catalog]: This master's current role is: LEADER
I20260929 01:56:37.054508 1067 sys_catalog.cc:458] T 00000000000000000000000000000000 P f8484e57b9504f259fc620b645926502 [sys.catalog]: This master's current role is: LEADER
W20260929 01:56:37.284917 1059 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:37.285362 1059 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:37.285580 1059 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:56:37.285662 1059 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:37.296729 1059 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:37.297011 1059 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:56:37.308947 1059 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38211,127.0.71.85:36585,127.0.71.86:37235,127.0.71.87:32931,127.0.71.88:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:39647
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=37983
--tserver_master_addrs=127.0.71.126:40625,127.0.71.125:39489,127.0.71.124:40767,127.0.71.123:43997
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:37.310920 1059 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:37.313470 1059 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:37.326874 1080 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:37.327167 1081 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:37.327916 1083 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:37.328888 1059 server_base.cc:1061] running on GCE node
I20260929 01:56:37.329737 1059 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:37.331207 1059 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:37.334829 1059 hybrid_clock.cc:648] HybridClock initialized: now 1790646997334537 us; error 144 us; skew 500 ppm
I20260929 01:56:37.335383 1059 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:37.338618 1059 webserver.cc:533] Webserver started at http://127.0.71.65:37983/ using document root <none> and password file <none>
I20260929 01:56:37.339704 1059 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:37.339875 1059 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:37.349287 1059 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 01:56:37.353554 1089 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:37.355619 1059 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260929 01:56:37.355921 1059 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "075b7745e5904472960152046484ddf5"
format_stamp: "Formatted at 2026-09-29 01:56:33 on dist-test-slave-vz23"
I20260929 01:56:37.356868 1059 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:37.394963 1059 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:37.396028 1059 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:37.396377 1059 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:37.397598 1059 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:56:37.400053 1059 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:56:37.400241 1059 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:37.400406 1059 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:56:37.400534 1059 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
W20260929 01:56:37.400995 1059 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:56:37.401156 1059 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:56:37.401255 1059 fault_injection.cc:59] injecting fault for kudu (pid 1059): FLAGS_fault_before_start (process will exit)
I20260929 01:56:37.418627 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:39647
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:40625,127.0.71.125:39489,127.0.71.124:40767
--webserver_port=37983
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:38211,127.0.71.85:36585,127.0.71.86:37235,127.0.71.87:32931,127.0.71.88:40527
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:40625,127.0.71.125:39489,127.0.71.124:40767,127.0.71.123:43997 with env {}
W20260929 01:56:37.861212 1097 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:37.861629 1097 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:37.861793 1097 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:37.872747 1097 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:37.873036 1097 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:56:37.886080 1097 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:38211,127.0.71.85:36585,127.0.71.86:37235,127.0.71.87:32931,127.0.71.88:40527
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:39647
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=37983
--tserver_master_addrs=127.0.71.126:40625,127.0.71.125:39489,127.0.71.124:40767,127.0.71.123:43997
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:37.888254 1097 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:37.890796 1097 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:37.904269 1102 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:37.904258 1103 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:37.905205 1105 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:37.905889 1097 server_base.cc:1061] running on GCE node
I20260929 01:56:37.906702 1097 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:56:37.908403 1097 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:56:37.912017 1097 hybrid_clock.cc:648] HybridClock initialized: now 1790646997911761 us; error 170 us; skew 500 ppm
I20260929 01:56:37.912719 1097 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:37.915881 1097 webserver.cc:533] Webserver started at http://127.0.71.65:37983/ using document root <none> and password file <none>
I20260929 01:56:37.917008 1097 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:37.917197 1097 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:37.925519 1097 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.002s
I20260929 01:56:37.929493 1111 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:37.931756 1097 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:56:37.932112 1097 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "075b7745e5904472960152046484ddf5"
format_stamp: "Formatted at 2026-09-29 01:56:33 on dist-test-slave-vz23"
I20260929 01:56:37.933216 1097 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:37.962981 1097 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:37.964092 1097 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:37.964488 1097 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:37.965673 1097 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:56:37.968000 1097 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:56:37.968159 1097 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:37.968333 1097 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:56:37.968410 1097 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:38.013365 1097 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:39647
I20260929 01:56:38.013432 1223 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:39647 every 8 connection(s)
I20260929 01:56:38.015420 1097 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:56:38.016924 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 1097
I20260929 01:56:38.017285 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 1097
I20260929 01:56:38.066025 1225 heartbeater.cc:344] Connected to a master server at 127.0.71.124:40767
I20260929 01:56:38.066561 1225 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:38.067895 1224 heartbeater.cc:344] Connected to a master server at 127.0.71.123:43997
I20260929 01:56:38.068094 1224 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:38.068945 1226 heartbeater.cc:344] Connected to a master server at 127.0.71.126:40625
I20260929 01:56:38.069967 1226 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:38.072419 1224 heartbeater.cc:507] Master 127.0.71.123:43997 requested a full tablet report, sending...
I20260929 01:56:38.072976 1228 heartbeater.cc:344] Connected to a master server at 127.0.71.125:39489
I20260929 01:56:38.073354 1228 heartbeater.cc:461] Registering TS with master...
I20260929 01:56:38.073367 1225 heartbeater.cc:507] Master 127.0.71.124:40767 requested a full tablet report, sending...
I20260929 01:56:38.074280 1228 heartbeater.cc:507] Master 127.0.71.125:39489 requested a full tablet report, sending...
I20260929 01:56:38.075255 1226 heartbeater.cc:507] Master 127.0.71.126:40625 requested a full tablet report, sending...
I20260929 01:56:38.075527 32643 ts_manager.cc:194] Re-registered known tserver with Master: 075b7745e5904472960152046484ddf5 (127.0.71.65:39647)
I20260929 01:56:38.075670 32716 ts_manager.cc:194] Re-registered known tserver with Master: 075b7745e5904472960152046484ddf5 (127.0.71.65:39647)
I20260929 01:56:38.075875 624 ts_manager.cc:194] Re-registered known tserver with Master: 075b7745e5904472960152046484ddf5 (127.0.71.65:39647)
I20260929 01:56:38.077982 32643 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:52951
I20260929 01:56:38.080003 1219 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:38.081885 851 ts_manager.cc:194] Registered new tserver with Master: 075b7745e5904472960152046484ddf5 (127.0.71.65:39647)
I20260929 01:56:38.181977 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 1097
I20260929 01:56:38.201131 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 820
I20260929 01:56:38.215585 881 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:38.339919 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 820
I20260929 01:56:38.367357 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 32611
I20260929 01:56:38.369030 32673 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:38.494422 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 32611
I20260929 01:56:38.520167 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 32680
I20260929 01:56:38.521919 32746 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:38.645448 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 32680
I20260929 01:56:38.669901 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 485
I20260929 01:56:38.671487 654 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:56:38.799887 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 485
2026-09-29T01:56:38Z chronyd exiting
2026-09-29T01:56:38Z chronyd exiting
2026-09-29T01:56:38Z chronyd exiting
2026-09-29T01:56:38Z chronyd exiting
2026-09-29T01:56:38Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestAddMaster/2 (9163 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/3
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/security.properties
2026-09-29 01:56:47: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260929 01:56:58.884982 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:41459
--webserver_interface=127.0.71.126
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:41459,127.0.71.125:38797,127.0.71.124:36653 with env {}
W20260929 01:56:59.315636 1523 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:59.316139 1523 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:59.316287 1523 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:59.325819 1523 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:56:59.325984 1523 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:59.326107 1523 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:56:59.326192 1523 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:56:59.337888 1523 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:41459,127.0.71.125:38797,127.0.71.124:36653
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:41459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:59.339782 1523 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:59.342096 1523 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:59.353071 1529 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:59.353039 1528 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:59.353113 1531 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:59.353816 1523 server_base.cc:1061] running on GCE node
I20260929 01:56:59.354895 1523 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:56:59.355091 1523 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:56:59.355171 1523 hybrid_clock.cc:648] HybridClock initialized: now 1790647019355150 us; error 0 us; skew 500 ppm
I20260929 01:56:59.355508 1523 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:59.359400 1523 webserver.cc:533] Webserver started at http://127.0.71.126:34645/ using document root <none> and password file <none>
I20260929 01:56:59.360536 1523 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:59.360702 1523 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:59.361104 1523 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:56:59.364254 1523 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "fd0047e5034a4f1483071751913abfaf"
format_stamp: "Formatted at 2026-09-29 01:56:59 on dist-test-slave-vz23"
I20260929 01:56:59.365527 1523 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "fd0047e5034a4f1483071751913abfaf"
format_stamp: "Formatted at 2026-09-29 01:56:59 on dist-test-slave-vz23"
I20260929 01:56:59.373621 1523 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260929 01:56:59.378607 1536 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:59.381213 1523 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.003s
I20260929 01:56:59.381455 1523 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "fd0047e5034a4f1483071751913abfaf"
format_stamp: "Formatted at 2026-09-29 01:56:59 on dist-test-slave-vz23"
I20260929 01:56:59.381699 1523 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:59.401280 1523 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:59.402344 1523 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:59.402645 1523 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:59.428783 1587 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:41459 every 8 connection(s)
I20260929 01:56:59.429004 1523 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:41459
I20260929 01:56:59.431298 1523 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:56:59.437117 1588 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:56:59.439594 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 1523
I20260929 01:56:59.439836 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:56:59.441295 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:38797
--webserver_interface=127.0.71.125
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:41459,127.0.71.125:38797,127.0.71.124:36653 with env {}
I20260929 01:56:59.447135 1588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:56:59.470042 1588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:56:59.471683 1537 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.125:38797: connect: Connection refused (error 111)
W20260929 01:56:59.474416 1588 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:38797: Network error: Client connection negotiation failed: client connection to 127.0.71.125:38797: connect: Connection refused (error 111)
I20260929 01:56:59.524042 1588 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } attempt: 1
W20260929 01:56:59.526917 1588 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:38797: Network error: Client connection negotiation failed: client connection to 127.0.71.125:38797: connect: Connection refused (error 111)
I20260929 01:56:59.595322 1588 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } attempt: 2
W20260929 01:56:59.598089 1588 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:38797: Network error: Client connection negotiation failed: client connection to 127.0.71.125:38797: connect: Connection refused (error 111)
I20260929 01:56:59.689463 1588 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } attempt: 3
W20260929 01:56:59.691869 1588 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:38797: Network error: Client connection negotiation failed: client connection to 127.0.71.125:38797: connect: Connection refused (error 111)
I20260929 01:56:59.835278 1588 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } attempt: 4
W20260929 01:56:59.838135 1588 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:38797: Network error: Client connection negotiation failed: client connection to 127.0.71.125:38797: connect: Connection refused (error 111)
W20260929 01:56:59.875185 1590 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:56:59.875654 1590 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:56:59.875742 1590 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:56:59.885641 1590 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:56:59.885787 1590 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:56:59.885864 1590 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:56:59.885921 1590 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:56:59.897574 1590 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:41459,127.0.71.125:38797,127.0.71.124:36653
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:38797
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:56:59.899665 1590 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:56:59.902086 1590 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:56:59.913257 1598 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:59.913226 1599 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:56:59.913618 1601 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:56:59.914390 1590 server_base.cc:1061] running on GCE node
I20260929 01:56:59.915498 1590 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:56:59.915699 1590 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:56:59.915788 1590 hybrid_clock.cc:648] HybridClock initialized: now 1790647019915763 us; error 0 us; skew 500 ppm
I20260929 01:56:59.916183 1590 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:56:59.920392 1590 webserver.cc:533] Webserver started at http://127.0.71.125:32969/ using document root <none> and password file <none>
I20260929 01:56:59.921561 1590 fs_manager.cc:362] Metadata directory not provided
I20260929 01:56:59.921723 1590 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:56:59.922176 1590 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:56:59.925414 1590 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-1/data/instance:
uuid: "c4adeb24a60248c587ab8d0dcecdde10"
format_stamp: "Formatted at 2026-09-29 01:56:59 on dist-test-slave-vz23"
I20260929 01:56:59.926481 1590 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-1/wal/instance:
uuid: "c4adeb24a60248c587ab8d0dcecdde10"
format_stamp: "Formatted at 2026-09-29 01:56:59 on dist-test-slave-vz23"
I20260929 01:56:59.934022 1590 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.002s
I20260929 01:56:59.938598 1606 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:56:59.941154 1590 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 01:56:59.941370 1590 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "c4adeb24a60248c587ab8d0dcecdde10"
format_stamp: "Formatted at 2026-09-29 01:56:59 on dist-test-slave-vz23"
I20260929 01:56:59.941670 1590 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:56:59.958983 1590 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:56:59.960116 1590 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:56:59.960475 1590 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:56:59.985239 1590 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:38797
I20260929 01:56:59.985390 1657 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:38797 every 8 connection(s)
I20260929 01:56:59.987238 1590 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:56:59.990978 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 1590
I20260929 01:56:59.991225 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:56:59.992285 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:36653
--webserver_interface=127.0.71.124
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:41459,127.0.71.125:38797,127.0.71.124:36653 with env {}
I20260929 01:56:59.994715 1658 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:57:00.002192 1658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:57:00.029517 1658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:57:00.041872 1658 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:57:00.043582 1609 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36653: connect: Connection refused (error 111)
W20260929 01:57:00.046077 1658 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36653: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36653: connect: Connection refused (error 111)
I20260929 01:57:00.095522 1658 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } attempt: 1
W20260929 01:57:00.098433 1658 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36653: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36653: connect: Connection refused (error 111)
I20260929 01:57:00.137564 1588 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } attempt: 5
I20260929 01:57:00.146631 1588 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:57:00.149220 1588 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36653: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36653: connect: Connection refused (error 111)
I20260929 01:57:00.166721 1658 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } attempt: 2
W20260929 01:57:00.169207 1658 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36653: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36653: connect: Connection refused (error 111)
I20260929 01:57:00.200572 1588 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } attempt: 1
W20260929 01:57:00.203094 1588 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36653: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36653: connect: Connection refused (error 111)
I20260929 01:57:00.260564 1658 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } attempt: 3
W20260929 01:57:00.263371 1658 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36653: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36653: connect: Connection refused (error 111)
I20260929 01:57:00.271422 1588 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } attempt: 2
W20260929 01:57:00.273845 1588 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36653: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36653: connect: Connection refused (error 111)
I20260929 01:57:00.380244 1588 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } attempt: 3
W20260929 01:57:00.383543 1588 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36653: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36653: connect: Connection refused (error 111)
I20260929 01:57:00.406720 1658 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } attempt: 4
W20260929 01:57:00.410357 1658 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36653: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36653: connect: Connection refused (error 111)
W20260929 01:57:00.420222 1660 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:57:00.420773 1660 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:57:00.420914 1660 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:57:00.430223 1660 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:57:00.430387 1660 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:57:00.430505 1660 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:57:00.430584 1660 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:57:00.441839 1660 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:41459,127.0.71.125:38797,127.0.71.124:36653
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:36653
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:57:00.443796 1660 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:57:00.446274 1660 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:57:00.457711 1669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:00.457746 1672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:00.457792 1670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:57:00.458972 1660 server_base.cc:1061] running on GCE node
I20260929 01:57:00.460012 1660 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:57:00.460208 1660 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:57:00.460289 1660 hybrid_clock.cc:648] HybridClock initialized: now 1790647020460270 us; error 0 us; skew 500 ppm
I20260929 01:57:00.460739 1660 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:57:00.465412 1660 webserver.cc:533] Webserver started at http://127.0.71.124:36269/ using document root <none> and password file <none>
I20260929 01:57:00.466616 1660 fs_manager.cc:362] Metadata directory not provided
I20260929 01:57:00.466770 1660 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:57:00.467664 1660 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:57:00.471357 1660 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-2/data/instance:
uuid: "87988700bf3342c7b8190f003fdd6524"
format_stamp: "Formatted at 2026-09-29 01:57:00 on dist-test-slave-vz23"
I20260929 01:57:00.472379 1660 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-2/wal/instance:
uuid: "87988700bf3342c7b8190f003fdd6524"
format_stamp: "Formatted at 2026-09-29 01:57:00 on dist-test-slave-vz23"
I20260929 01:57:00.479764 1660 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.002s sys 0.003s
I20260929 01:57:00.484965 1677 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:00.487149 1660 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260929 01:57:00.487499 1660 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "87988700bf3342c7b8190f003fdd6524"
format_stamp: "Formatted at 2026-09-29 01:57:00 on dist-test-slave-vz23"
I20260929 01:57:00.487833 1660 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:57:00.512043 1660 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:57:00.513456 1660 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:57:00.513823 1660 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:57:00.541460 1660 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:36653
I20260929 01:57:00.541538 1728 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:36653 every 8 connection(s)
I20260929 01:57:00.543608 1660 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:57:00.550032 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 1660
I20260929 01:57:00.550305 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:57:00.550535 1729 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:57:00.561312 1588 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } attempt: 4
I20260929 01:57:00.561952 1729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:57:00.579401 1729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:57:00.588156 1588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf: Bootstrap starting.
I20260929 01:57:00.588748 1729 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:57:00.593103 1588 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf: Neither blocks nor log segments found. Creating new log.
I20260929 01:57:00.595614 1588 log.cc:826] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf: Log is configured to *not* fsync() on all Append() calls
I20260929 01:57:00.599661 1588 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf: No bootstrap required, opened a new log
I20260929 01:57:00.601923 1729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524: Bootstrap starting.
I20260929 01:57:00.606935 1729 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524: Neither blocks nor log segments found. Creating new log.
I20260929 01:57:00.607522 1588 raft_consensus.cc:359] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } }
I20260929 01:57:00.607915 1588 raft_consensus.cc:385] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:57:00.608069 1588 raft_consensus.cc:740] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd0047e5034a4f1483071751913abfaf, State: Initialized, Role: FOLLOWER
I20260929 01:57:00.608425 1729 log.cc:826] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524: Log is configured to *not* fsync() on all Append() calls
I20260929 01:57:00.608935 1588 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } }
I20260929 01:57:00.610201 1738 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } }
I20260929 01:57:00.610742 1738 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:00.612043 1588 sys_catalog.cc:565] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:57:00.612709 1729 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524: No bootstrap required, opened a new log
I20260929 01:57:00.620098 1729 raft_consensus.cc:359] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } }
I20260929 01:57:00.620649 1729 raft_consensus.cc:385] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:57:00.620829 1729 raft_consensus.cc:740] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 87988700bf3342c7b8190f003fdd6524, State: Initialized, Role: FOLLOWER
I20260929 01:57:00.621762 1729 consensus_queue.cc:260] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } }
I20260929 01:57:00.624588 1750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } }
I20260929 01:57:00.625082 1750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:00.625969 1729 sys_catalog.cc:565] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:57:00.626260 1523 master_runner.cc:187] Error getting master registration for 127.0.71.125:38797: OK, instance_id { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" instance_seqno: 1790647019978449 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:57:00.637985 1523 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:57:00.640836 1749 catalog_manager.cc:1607] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:57:00.641105 1749 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 01:57:00.656078 1762 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:57:00.656359 1542 tablet.cc:2433] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:57:00.656529 1762 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:57:00.650275 1660 master_runner.cc:187] Error getting master registration for 127.0.71.125:38797: OK, instance_id { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" instance_seqno: 1790647019978449 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:57:00.659701 1660 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:57:00.709729 1658 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } attempt: 5
I20260929 01:57:00.731426 1658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10: Bootstrap starting.
I20260929 01:57:00.736783 1658 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10: Neither blocks nor log segments found. Creating new log.
I20260929 01:57:00.738407 1658 log.cc:826] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10: Log is configured to *not* fsync() on all Append() calls
I20260929 01:57:00.739007 1750 raft_consensus.cc:493] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:57:00.739357 1750 raft_consensus.cc:515] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } }
I20260929 01:57:00.741768 1750 leader_election.cc:290] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers fd0047e5034a4f1483071751913abfaf (127.0.71.126:41459), c4adeb24a60248c587ab8d0dcecdde10 (127.0.71.125:38797)
I20260929 01:57:00.743444 1563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87988700bf3342c7b8190f003fdd6524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd0047e5034a4f1483071751913abfaf" is_pre_election: true
I20260929 01:57:00.744095 1563 raft_consensus.cc:2468] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 87988700bf3342c7b8190f003fdd6524 in term 0.
I20260929 01:57:00.745253 1678 leader_election.cc:304] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87988700bf3342c7b8190f003fdd6524, fd0047e5034a4f1483071751913abfaf; no voters:
I20260929 01:57:00.745817 1750 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:57:00.745987 1750 raft_consensus.cc:493] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:57:00.746116 1750 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:57:00.750845 1658 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10: No bootstrap required, opened a new log
I20260929 01:57:00.752556 1750 raft_consensus.cc:515] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } }
I20260929 01:57:00.756495 1563 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87988700bf3342c7b8190f003fdd6524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd0047e5034a4f1483071751913abfaf"
I20260929 01:57:00.756804 1563 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:57:00.762604 1750 leader_election.cc:290] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [CANDIDATE]: Term 1 election: Requested vote from peers fd0047e5034a4f1483071751913abfaf (127.0.71.126:41459), c4adeb24a60248c587ab8d0dcecdde10 (127.0.71.125:38797)
I20260929 01:57:00.763290 1563 raft_consensus.cc:2468] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87988700bf3342c7b8190f003fdd6524 in term 1.
I20260929 01:57:00.764081 1678 leader_election.cc:304] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 87988700bf3342c7b8190f003fdd6524, fd0047e5034a4f1483071751913abfaf; no voters:
I20260929 01:57:00.764710 1750 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:57:00.764829 1658 raft_consensus.cc:359] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } }
I20260929 01:57:00.765823 1658 raft_consensus.cc:385] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:57:00.766240 1658 raft_consensus.cc:740] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c4adeb24a60248c587ab8d0dcecdde10, State: Initialized, Role: FOLLOWER
W20260929 01:57:00.766674 1683 tablet.cc:2433] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:57:00.769490 1658 consensus_queue.cc:260] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } }
I20260929 01:57:00.771720 1750 raft_consensus.cc:697] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [term 1 LEADER]: Becoming Leader. State: Replica: 87988700bf3342c7b8190f003fdd6524, State: Running, Role: LEADER
I20260929 01:57:00.772835 1750 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } }
I20260929 01:57:00.781344 1766 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } }
I20260929 01:57:00.781790 1766 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:00.784952 1658 sys_catalog.cc:565] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:57:00.784917 1632 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87988700bf3342c7b8190f003fdd6524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4adeb24a60248c587ab8d0dcecdde10"
I20260929 01:57:00.785732 1632 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:57:00.790433 1632 raft_consensus.cc:2468] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 87988700bf3342c7b8190f003fdd6524 in term 1.
I20260929 01:57:00.791026 1633 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "87988700bf3342c7b8190f003fdd6524" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c4adeb24a60248c587ab8d0dcecdde10" is_pre_election: true
I20260929 01:57:00.790908 1765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87988700bf3342c7b8190f003fdd6524. Latest consensus state: current_term: 1 leader_uuid: "87988700bf3342c7b8190f003fdd6524" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } }
I20260929 01:57:00.791378 1765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: This master's current role is: LEADER
I20260929 01:57:00.791394 1633 raft_consensus.cc:2376] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10 [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate 87988700bf3342c7b8190f003fdd6524 in term 1. Re-sending same reply.
I20260929 01:57:00.801909 1769 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:57:00.806154 1769 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:57:00.833957 1633 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10 [term 1 FOLLOWER]: Refusing update from remote peer 87988700bf3342c7b8190f003fdd6524: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:57:00.838905 1765 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:57:00.852682 1563 raft_consensus.cc:1275] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [term 1 FOLLOWER]: Refusing update from remote peer 87988700bf3342c7b8190f003fdd6524: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:57:00.854387 1765 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:57:00.880743 1738 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87988700bf3342c7b8190f003fdd6524. Latest consensus state: current_term: 1 leader_uuid: "87988700bf3342c7b8190f003fdd6524" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } }
I20260929 01:57:00.881126 1738 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:00.881639 1766 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87988700bf3342c7b8190f003fdd6524. Latest consensus state: current_term: 1 leader_uuid: "87988700bf3342c7b8190f003fdd6524" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } }
I20260929 01:57:00.881938 1766 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:00.890192 1772 mvcc.cc:204] Tried to move back new op lower bound from 7334490197319761920 to 7334490197118324736. Current Snapshot: MvccSnapshot[applied={T|T < 7334490197319761920}]
I20260929 01:57:00.909801 1766 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87988700bf3342c7b8190f003fdd6524" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } }
I20260929 01:57:00.910223 1766 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:00.918370 1765 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87988700bf3342c7b8190f003fdd6524" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } }
I20260929 01:57:00.918856 1765 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: This master's current role is: LEADER
I20260929 01:57:00.922451 1750 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87988700bf3342c7b8190f003fdd6524" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } }
I20260929 01:57:00.923111 1750 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: This master's current role is: LEADER
I20260929 01:57:00.927435 1769 catalog_manager.cc:1396] Generated new cluster ID: 35495148a72749dd9d4814042afd810d
I20260929 01:57:00.927634 1769 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:57:00.930962 1738 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87988700bf3342c7b8190f003fdd6524" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } }
I20260929 01:57:00.931254 1738 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:01.006928 1769 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 01:57:01.009207 1769 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:57:01.030644 1788 catalog_manager.cc:1308] Loaded cluster ID: 35495148a72749dd9d4814042afd810d
I20260929 01:57:01.030833 1788 catalog_manager.cc:1601] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10: loading cluster ID for follower catalog manager: success
I20260929 01:57:01.035820 1788 catalog_manager.cc:1623] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10: acquiring CA information for follower catalog manager: success
W20260929 01:57:01.038028 1788 catalog_manager.cc:896] Not found: no TSK found in the system table: failed to prepare follower catalog manager, will retry
I20260929 01:57:01.038544 1769 catalog_manager.cc:6219] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524: Generated new TSK 0
I20260929 01:57:01.040002 1769 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:57:01.062737 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:41459,127.0.71.125:38797,127.0.71.124:36653
--time_source=system_unsync with env {}
I20260929 01:57:01.643921 1749 catalog_manager.cc:1308] Loaded cluster ID: 35495148a72749dd9d4814042afd810d
I20260929 01:57:01.644080 1749 catalog_manager.cc:1601] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf: loading cluster ID for follower catalog manager: success
I20260929 01:57:01.649080 1749 catalog_manager.cc:1623] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf: acquiring CA information for follower catalog manager: success
I20260929 01:57:01.652840 1749 catalog_manager.cc:1651] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 01:57:01.676324 1790 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:57:01.676870 1790 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:57:01.677029 1790 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:57:01.686847 1790 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:57:01.687101 1790 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:57:01.698347 1790 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:41459,127.0.71.125:38797,127.0.71.124:36653
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:57:01.700666 1790 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:57:01.703161 1790 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:57:01.719867 1795 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:01.719753 1796 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:01.722162 1798 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:57:01.722183 1790 server_base.cc:1061] running on GCE node
I20260929 01:57:01.723085 1790 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:57:01.723301 1790 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:57:01.723439 1790 hybrid_clock.cc:648] HybridClock initialized: now 1790647021723412 us; error 0 us; skew 500 ppm
I20260929 01:57:01.723806 1790 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:57:01.727211 1790 webserver.cc:533] Webserver started at http://127.0.71.65:37423/ using document root <none> and password file <none>
I20260929 01:57:01.728402 1790 fs_manager.cc:362] Metadata directory not provided
I20260929 01:57:01.728688 1790 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:57:01.729156 1790 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:57:01.733035 1790 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "09898dc208db41b8aacc96c92d4708f6"
format_stamp: "Formatted at 2026-09-29 01:57:01 on dist-test-slave-vz23"
I20260929 01:57:01.734160 1790 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "09898dc208db41b8aacc96c92d4708f6"
format_stamp: "Formatted at 2026-09-29 01:57:01 on dist-test-slave-vz23"
I20260929 01:57:01.743115 1790 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.005s sys 0.004s
I20260929 01:57:01.748189 1803 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:01.750476 1790 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260929 01:57:01.750794 1790 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "09898dc208db41b8aacc96c92d4708f6"
format_stamp: "Formatted at 2026-09-29 01:57:01 on dist-test-slave-vz23"
I20260929 01:57:01.751101 1790 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:57:01.781118 1790 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:57:01.782294 1790 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:57:01.782660 1790 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:57:01.783818 1790 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:57:01.785946 1790 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:57:01.786118 1790 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:01.786265 1790 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:57:01.786360 1790 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:01.828575 1790 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:41935
I20260929 01:57:01.828771 1915 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:41935 every 8 connection(s)
I20260929 01:57:01.830509 1790 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:57:01.836807 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 1790
I20260929 01:57:01.837090 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
I20260929 01:57:01.867787 1916 heartbeater.cc:344] Connected to a master server at 127.0.71.124:36653
I20260929 01:57:01.867871 1917 heartbeater.cc:344] Connected to a master server at 127.0.71.126:41459
I20260929 01:57:01.868342 1916 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:01.868337 1917 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:01.870018 1917 heartbeater.cc:507] Master 127.0.71.126:41459 requested a full tablet report, sending...
I20260929 01:57:01.870051 1916 heartbeater.cc:507] Master 127.0.71.124:36653 requested a full tablet report, sending...
I20260929 01:57:01.870549 1918 heartbeater.cc:344] Connected to a master server at 127.0.71.125:38797
I20260929 01:57:01.870730 1918 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:01.871430 1918 heartbeater.cc:507] Master 127.0.71.125:38797 requested a full tablet report, sending...
I20260929 01:57:01.873127 1694 ts_manager.cc:194] Registered new tserver with Master: 09898dc208db41b8aacc96c92d4708f6 (127.0.71.65:41935)
I20260929 01:57:01.873482 1553 ts_manager.cc:194] Registered new tserver with Master: 09898dc208db41b8aacc96c92d4708f6 (127.0.71.65:41935)
I20260929 01:57:01.873572 1623 ts_manager.cc:194] Registered new tserver with Master: 09898dc208db41b8aacc96c92d4708f6 (127.0.71.65:41935)
I20260929 01:57:01.875344 1694 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:39923
I20260929 01:57:01.883276 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:57:01.886335 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.123:37063
--webserver_interface=127.0.71.123
--webserver_port=0
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:41459,127.0.71.125:38797,127.0.71.124:36653,127.0.71.123:37063 with env {}
I20260929 01:57:02.042016 1788 catalog_manager.cc:1651] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 01:57:02.314702 1926 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:57:02.315121 1926 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:57:02.315207 1926 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:57:02.325387 1926 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:57:02.325524 1926 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:57:02.325588 1926 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:57:02.325631 1926 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:57:02.337534 1926 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:41459,127.0.71.125:38797,127.0.71.124:36653,127.0.71.123:37063
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.123:37063
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.0.71.123
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:57:02.339524 1926 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:57:02.341656 1926 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:57:02.352002 1934 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:02.352355 1932 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:02.352072 1931 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:57:02.354120 1926 server_base.cc:1061] running on GCE node
I20260929 01:57:02.355232 1926 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:57:02.355453 1926 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:57:02.355530 1926 hybrid_clock.cc:648] HybridClock initialized: now 1790647022355507 us; error 0 us; skew 500 ppm
I20260929 01:57:02.355909 1926 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:57:02.359984 1926 webserver.cc:533] Webserver started at http://127.0.71.123:35213/ using document root <none> and password file <none>
I20260929 01:57:02.361044 1926 fs_manager.cc:362] Metadata directory not provided
I20260929 01:57:02.361174 1926 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:57:02.361550 1926 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:57:02.364653 1926 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-3/data/instance:
uuid: "3405de609e5643159e835f3a17f44097"
format_stamp: "Formatted at 2026-09-29 01:57:02 on dist-test-slave-vz23"
I20260929 01:57:02.365738 1926 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-3/wal/instance:
uuid: "3405de609e5643159e835f3a17f44097"
format_stamp: "Formatted at 2026-09-29 01:57:02 on dist-test-slave-vz23"
I20260929 01:57:02.373752 1926 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.002s
I20260929 01:57:02.379437 1939 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:02.381671 1926 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260929 01:57:02.382009 1926 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "3405de609e5643159e835f3a17f44097"
format_stamp: "Formatted at 2026-09-29 01:57:02 on dist-test-slave-vz23"
I20260929 01:57:02.382339 1926 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:57:02.405069 1926 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:57:02.406152 1926 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:57:02.406513 1926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:57:02.433172 1926 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.123:37063
I20260929 01:57:02.435307 1926 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
I20260929 01:57:02.433154 1990 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.123:37063 every 8 connection(s)
I20260929 01:57:02.440936 1991 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:57:02.441241 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 1926
I20260929 01:57:02.453289 1991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:57:02.468896 1991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:57:02.476805 1991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:57:02.484875 1991 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.123" port: 37063 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:57:02.498646 1991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097: Bootstrap starting.
I20260929 01:57:02.502876 1991 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097: Neither blocks nor log segments found. Creating new log.
I20260929 01:57:02.504750 1991 log.cc:826] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097: Log is configured to *not* fsync() on all Append() calls
I20260929 01:57:02.508414 1991 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097: No bootstrap required, opened a new log
I20260929 01:57:02.515061 1991 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 37063 } }
I20260929 01:57:02.515581 1991 raft_consensus.cc:385] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:57:02.515712 1991 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 3405de609e5643159e835f3a17f44097, State: Initialized, Role: FOLLOWER
I20260929 01:57:02.516691 1991 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 37063 } }
I20260929 01:57:02.517932 2000 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 37063 } } }
I20260929 01:57:02.518262 2000 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:02.519706 1991 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:57:02.533088 2011 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:57:02.533303 2011 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:57:02.545070 1926 master_runner.cc:429] Detected that this master 3405de609e5643159e835f3a17f44097 is joining an existing cluster
I20260929 01:57:02.545226 1926 catalog_manager.cc:1037] Shutting down the Catalog Manager
I20260929 01:57:02.546525 1926 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260929 01:57:02.547206 1926 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260929 01:57:02.547629 1926 tablet_replica.cc:333] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097: stopping tablet replica
I20260929 01:57:02.553378 1926 master_runner.cc:434] Initiating AddMaster RPC to add 127.0.71.123:37063
I20260929 01:57:02.611369 1694 catalog_manager.cc:7351] Initiating ChangeConfig request to add master 127.0.71.123:37063: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "3405de609e5643159e835f3a17f44097"
member_type: NON_VOTER
last_known_addr {
host: "127.0.71.123"
port: 37063
}
attrs {
promote: true
}
}
dest_uuid: "87988700bf3342c7b8190f003fdd6524"
cas_config_opid_index: -1
I20260929 01:57:02.613029 1694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: true } }
I20260929 01:57:02.616950 1563 raft_consensus.cc:1275] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [term 1 FOLLOWER]: Refusing update from remote peer 87988700bf3342c7b8190f003fdd6524: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 01:57:02.617720 1633 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10 [term 1 FOLLOWER]: Refusing update from remote peer 87988700bf3342c7b8190f003fdd6524: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20260929 01:57:02.618235 1681 consensus_peers.cc:597] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 -> Peer 3405de609e5643159e835f3a17f44097 (127.0.71.123:37063): Couldn't send request to peer 3405de609e5643159e835f3a17f44097. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260929 01:57:02.618252 1781 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:57:02.618742 2023 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:57:02.626353 1781 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3405de609e5643159e835f3a17f44097 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: true } } }
I20260929 01:57:02.630427 1563 raft_consensus.cc:2955] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3405de609e5643159e835f3a17f44097 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: true } } }
I20260929 01:57:02.631645 1781 catalog_manager.cc:7355] Successfully completed master ChangeConfig request to add master 127.0.71.123:37063
I20260929 01:57:02.632359 1632 raft_consensus.cc:2955] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 3405de609e5643159e835f3a17f44097 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: true } } }
I20260929 01:57:02.637560 1926 master.cc:562] Master@127.0.71.123:37063 shutting down...
I20260929 01:57:02.645936 2028 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87988700bf3342c7b8190f003fdd6524" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: true } } }
I20260929 01:57:02.646607 2028 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:02.648387 2023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "87988700bf3342c7b8190f003fdd6524" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: true } } }
I20260929 01:57:02.648754 2023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: This master's current role is: LEADER
I20260929 01:57:02.649811 2023 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87988700bf3342c7b8190f003fdd6524" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: true } } }
I20260929 01:57:02.649930 1781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87988700bf3342c7b8190f003fdd6524" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: true } } }
I20260929 01:57:02.650094 2023 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: This master's current role is: LEADER
I20260929 01:57:02.650127 1781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: This master's current role is: LEADER
I20260929 01:57:02.647874 2027 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87988700bf3342c7b8190f003fdd6524" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: true } } }
I20260929 01:57:02.651139 2027 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:02.676155 1926 master.cc:584] Master@127.0.71.123:37063 shutdown complete.
I20260929 01:57:02.676273 1926 master_runner.cc:306] Clearing existing system tablet
I20260929 01:57:02.676769 1926 fs_manager.cc:362] Metadata directory not provided
I20260929 01:57:02.676913 1926 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:57:02.680984 1926 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:57:02.683197 2032 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:02.683736 1926 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:57:02.684082 1926 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260929 01:57:02.684221 1926 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "3405de609e5643159e835f3a17f44097"
format_stamp: "Formatted at 2026-09-29 01:57:02 on dist-test-slave-vz23"
I20260929 01:57:02.689790 1926 ts_tablet_manager.cc:1962] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:57:02.698889 1926 ts_tablet_manager.cc:1975] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 01:57:02.699136 1926 log.cc:1199] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260929 01:57:02.699707 1926 ts_tablet_manager.cc:1996] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097: Deleting consensus metadata
I20260929 01:57:02.700047 1926 master_runner.cc:316] Copying system tablet from 127.0.71.124:36653
I20260929 01:57:02.703514 1926 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097: tablet copy: Beginning tablet copy session from remote peer at address 127.0.71.124:36653
I20260929 01:57:02.714263 1714 tablet_copy_service.cc:140] P 87988700bf3342c7b8190f003fdd6524: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 3405de609e5643159e835f3a17f44097 ({username='slave'} at 127.0.0.1:32798)
I20260929 01:57:02.714690 1714 tablet_copy_service.cc:161] P 87988700bf3342c7b8190f003fdd6524: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 3405de609e5643159e835f3a17f44097 at {username='slave'} at 127.0.0.1:32798: session id = 3405de609e5643159e835f3a17f44097-00000000000000000000000000000000
I20260929 01:57:02.718349 1714 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 01:57:02.721652 1926 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 01:57:02.730365 1926 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097: tablet copy: Starting download of 0 data blocks...
I20260929 01:57:02.730803 1926 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097: tablet copy: Starting download of 1 WAL segments...
I20260929 01:57:02.736027 1926 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 01:57:02.742376 1714 tablet_copy_service.cc:342] P 87988700bf3342c7b8190f003fdd6524: Request end of tablet copy session 3405de609e5643159e835f3a17f44097-00000000000000000000000000000000 received from {username='slave'} at 127.0.0.1:32798
I20260929 01:57:02.742710 1714 tablet_copy_service.cc:434] P 87988700bf3342c7b8190f003fdd6524: ending tablet copy session 3405de609e5643159e835f3a17f44097-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 3405de609e5643159e835f3a17f44097
I20260929 01:57:02.756911 1926 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:57:02.760273 2039 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:02.760648 2040 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:02.765893 2042 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:57:02.767160 1926 server_base.cc:1061] running on GCE node
I20260929 01:57:02.767628 1926 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:57:02.767762 1926 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:57:02.767850 1926 hybrid_clock.cc:648] HybridClock initialized: now 1790647022767822 us; error 0 us; skew 500 ppm
I20260929 01:57:02.768093 1926 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:57:02.769866 1926 webserver.cc:533] Webserver started at http://127.0.71.123:38613/ using document root <none> and password file <none>
I20260929 01:57:02.770305 1926 fs_manager.cc:362] Metadata directory not provided
I20260929 01:57:02.770412 1926 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:57:02.773994 1926 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:57:02.776118 2047 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:02.776808 1926 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260929 01:57:02.776983 1926 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "3405de609e5643159e835f3a17f44097"
format_stamp: "Formatted at 2026-09-29 01:57:02 on dist-test-slave-vz23"
I20260929 01:57:02.777192 1926 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:57:02.793476 1926 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:57:02.794422 1926 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:57:02.817137 1926 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.123:37063
I20260929 01:57:02.817178 2098 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.123:37063 every 8 connection(s)
I20260929 01:57:02.818388 1926 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
I20260929 01:57:02.823135 2099 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:57:02.826081 2099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097: Bootstrap starting.
I20260929 01:57:02.851729 2099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:57:02.852418 2099 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097: Bootstrap complete.
I20260929 01:57:02.853793 2099 raft_consensus.cc:359] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: true } }
I20260929 01:57:02.854104 2099 raft_consensus.cc:740] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 3405de609e5643159e835f3a17f44097, State: Initialized, Role: LEARNER
I20260929 01:57:02.854434 2099 consensus_queue.cc:260] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: true } }
I20260929 01:57:02.855571 2103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: true } } }
I20260929 01:57:02.855938 2103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097 [sys.catalog]: This master's current role is: LEARNER
I20260929 01:57:02.855952 2099 sys_catalog.cc:565] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:57:02.866896 2114 catalog_manager.cc:1308] Loaded cluster ID: 35495148a72749dd9d4814042afd810d
I20260929 01:57:02.867031 2114 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097: loading cluster ID for follower catalog manager: success
I20260929 01:57:02.870699 2114 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097: acquiring CA information for follower catalog manager: success
I20260929 01:57:02.873558 2114 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:57:02.879921 1916 heartbeater.cc:499] Master 127.0.71.124:36653 was elected leader, sending a full tablet report...
I20260929 01:57:02.898483 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 1790
I20260929 01:57:02.909003 1911 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:57:03.011251 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 1790
I20260929 01:57:03.032606 2074 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260929 01:57:03.033788 2103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87988700bf3342c7b8190f003fdd6524. Latest consensus state: current_term: 1 leader_uuid: "87988700bf3342c7b8190f003fdd6524" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: true } } }
I20260929 01:57:03.034212 2103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097 [sys.catalog]: This master's current role is: LEARNER
I20260929 01:57:03.037358 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:41935
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:41459,127.0.71.125:38797,127.0.71.124:36653
--webserver_port=37423
--webserver_interface=127.0.71.65
--time_source=system_unsync
--tserver_master_addrs=127.0.71.126:41459,127.0.71.125:38797,127.0.71.124:36653,127.0.71.123:37063 with env {}
I20260929 01:57:03.469290 1781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87988700bf3342c7b8190f003fdd6524" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: true } } }
I20260929 01:57:03.469623 1781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: This master's current role is: LEADER
I20260929 01:57:03.476361 2121 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524: attempting to promote NON_VOTER 3405de609e5643159e835f3a17f44097 to VOTER
I20260929 01:57:03.477876 2121 consensus_queue.cc:237] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: false } }
I20260929 01:57:03.481385 2074 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097 [term 1 LEARNER]: Refusing update from remote peer 87988700bf3342c7b8190f003fdd6524: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:57:03.481477 1632 raft_consensus.cc:1275] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10 [term 1 FOLLOWER]: Refusing update from remote peer 87988700bf3342c7b8190f003fdd6524: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:57:03.481535 1563 raft_consensus.cc:1275] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [term 1 FOLLOWER]: Refusing update from remote peer 87988700bf3342c7b8190f003fdd6524: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:57:03.482340 2122 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:57:03.482841 1781 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:57:03.483971 2121 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260929 01:57:03.485105 2117 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:57:03.486238 2117 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:57:03.486446 2117 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260929 01:57:03.489140 1563 raft_consensus.cc:2955] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3405de609e5643159e835f3a17f44097 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: false } } }
I20260929 01:57:03.488967 1781 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3405de609e5643159e835f3a17f44097 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: false } } }
I20260929 01:57:03.496206 2121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "87988700bf3342c7b8190f003fdd6524" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: false } } }
I20260929 01:57:03.496852 2121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: This master's current role is: LEADER
I20260929 01:57:03.496214 1632 raft_consensus.cc:2955] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3405de609e5643159e835f3a17f44097 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: false } } }
I20260929 01:57:03.497241 2122 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87988700bf3342c7b8190f003fdd6524" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: false } } }
I20260929 01:57:03.497673 2122 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: This master's current role is: LEADER
I20260929 01:57:03.498042 1781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87988700bf3342c7b8190f003fdd6524" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: false } } }
I20260929 01:57:03.498297 1781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: This master's current role is: LEADER
I20260929 01:57:03.498598 1781 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87988700bf3342c7b8190f003fdd6524" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: false } } }
I20260929 01:57:03.498811 1781 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: This master's current role is: LEADER
I20260929 01:57:03.499879 2074 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 3405de609e5643159e835f3a17f44097 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: false } } }
I20260929 01:57:03.501832 2125 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87988700bf3342c7b8190f003fdd6524" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: false } } }
I20260929 01:57:03.502221 2125 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:57:03.503744 2117 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:57:03.504021 2117 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:57:03.504498 2126 sys_catalog.cc:455] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87988700bf3342c7b8190f003fdd6524" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: false } } }
I20260929 01:57:03.504797 2126 sys_catalog.cc:458] T 00000000000000000000000000000000 P c4adeb24a60248c587ab8d0dcecdde10 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:03.505155 2103 sys_catalog.cc:455] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "87988700bf3342c7b8190f003fdd6524" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: false } } }
I20260929 01:57:03.505424 2103 sys_catalog.cc:458] T 00000000000000000000000000000000 P 3405de609e5643159e835f3a17f44097 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:03.518219 2117 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:41935
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=37423
--tserver_master_addrs=127.0.71.126:41459,127.0.71.125:38797,127.0.71.124:36653,127.0.71.123:37063
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:57:03.520413 2117 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:57:03.522902 2117 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:57:03.536757 2129 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:03.536677 2128 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:03.537544 2131 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:57:03.537978 2117 server_base.cc:1061] running on GCE node
I20260929 01:57:03.538841 2117 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:57:03.539079 2117 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:57:03.539211 2117 hybrid_clock.cc:648] HybridClock initialized: now 1790647023539162 us; error 0 us; skew 500 ppm
I20260929 01:57:03.539628 2117 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:57:03.543246 2117 webserver.cc:533] Webserver started at http://127.0.71.65:37423/ using document root <none> and password file <none>
I20260929 01:57:03.544698 2117 fs_manager.cc:362] Metadata directory not provided
I20260929 01:57:03.544942 2117 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:57:03.554808 2117 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.000s sys 0.004s
I20260929 01:57:03.559682 2136 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:03.562247 2117 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260929 01:57:03.562476 2117 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "09898dc208db41b8aacc96c92d4708f6"
format_stamp: "Formatted at 2026-09-29 01:57:01 on dist-test-slave-vz23"
I20260929 01:57:03.563401 2117 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:57:03.585670 2117 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:57:03.586826 2117 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:57:03.587165 2117 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:57:03.588423 2117 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:57:03.590507 2117 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:57:03.590610 2117 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:03.590771 2117 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:57:03.590835 2117 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:03.632658 2117 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:41935
I20260929 01:57:03.632702 2248 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:41935 every 8 connection(s)
I20260929 01:57:03.634822 2117 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:57:03.638460 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 2117
I20260929 01:57:03.674644 2249 heartbeater.cc:344] Connected to a master server at 127.0.71.124:36653
I20260929 01:57:03.674826 2250 heartbeater.cc:344] Connected to a master server at 127.0.71.123:37063
I20260929 01:57:03.675410 2249 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:03.675431 2250 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:03.677089 2250 heartbeater.cc:507] Master 127.0.71.123:37063 requested a full tablet report, sending...
I20260929 01:57:03.677124 2249 heartbeater.cc:507] Master 127.0.71.124:36653 requested a full tablet report, sending...
I20260929 01:57:03.677707 2251 heartbeater.cc:344] Connected to a master server at 127.0.71.126:41459
I20260929 01:57:03.677877 2251 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:03.678581 2251 heartbeater.cc:507] Master 127.0.71.126:41459 requested a full tablet report, sending...
I20260929 01:57:03.680217 1694 ts_manager.cc:194] Re-registered known tserver with Master: 09898dc208db41b8aacc96c92d4708f6 (127.0.71.65:41935)
I20260929 01:57:03.681203 2064 ts_manager.cc:194] Registered new tserver with Master: 09898dc208db41b8aacc96c92d4708f6 (127.0.71.65:41935)
I20260929 01:57:03.681476 1694 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:52507
I20260929 01:57:03.681526 2252 heartbeater.cc:344] Connected to a master server at 127.0.71.125:38797
I20260929 01:57:03.681731 2252 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:03.682416 2252 heartbeater.cc:507] Master 127.0.71.125:38797 requested a full tablet report, sending...
I20260929 01:57:03.682826 1552 ts_manager.cc:194] Re-registered known tserver with Master: 09898dc208db41b8aacc96c92d4708f6 (127.0.71.65:41935)
I20260929 01:57:03.683884 1623 ts_manager.cc:194] Re-registered known tserver with Master: 09898dc208db41b8aacc96c92d4708f6 (127.0.71.65:41935)
I20260929 01:57:03.692375 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:57:03.708072 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:57:03.708269 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 1523
I20260929 01:57:03.709509 1583 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:57:03.830628 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 1523
I20260929 01:57:03.856133 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:41459
--webserver_interface=127.0.71.126
--webserver_port=34645
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:41459,127.0.71.125:38797,127.0.71.124:36653
--master_addresses=127.0.71.126:41459,127.0.71.125:38797,127.0.71.124:36653,127.0.71.123:37063 with env {}
W20260929 01:57:04.007354 1678 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.126:41459: connect: Connection refused (error 111)
W20260929 01:57:04.009181 1678 consensus_peers.cc:597] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 -> Peer fd0047e5034a4f1483071751913abfaf (127.0.71.126:41459): Couldn't send request to peer fd0047e5034a4f1483071751913abfaf. Status: Network error: Client connection negotiation failed: client connection to 127.0.71.126:41459: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260929 01:57:04.301307 2263 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:57:04.301898 2263 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:57:04.302085 2263 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:57:04.311555 2263 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:57:04.311754 2263 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:57:04.311875 2263 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:57:04.311965 2263 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:57:04.323452 2263 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:41459,127.0.71.125:38797,127.0.71.124:36653,127.0.71.123:37063
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:41459
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=34645
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:57:04.325582 2263 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:57:04.328012 2263 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:57:04.338661 2271 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:04.339504 2270 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:04.340757 2273 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:57:04.341697 2263 server_base.cc:1061] running on GCE node
I20260929 01:57:04.342738 2263 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:57:04.342969 2263 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:57:04.343079 2263 hybrid_clock.cc:648] HybridClock initialized: now 1790647024343053 us; error 0 us; skew 500 ppm
I20260929 01:57:04.343478 2263 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:57:04.347415 2263 webserver.cc:533] Webserver started at http://127.0.71.126:34645/ using document root <none> and password file <none>
I20260929 01:57:04.348495 2263 fs_manager.cc:362] Metadata directory not provided
I20260929 01:57:04.348692 2263 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:57:04.355945 2263 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.001s
I20260929 01:57:04.360662 2278 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:04.363060 2263 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260929 01:57:04.363394 2263 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "fd0047e5034a4f1483071751913abfaf"
format_stamp: "Formatted at 2026-09-29 01:56:59 on dist-test-slave-vz23"
I20260929 01:57:04.364360 2263 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:57:04.394035 2263 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:57:04.395341 2263 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:57:04.395731 2263 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:57:04.427889 2329 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:41459 every 8 connection(s)
I20260929 01:57:04.428061 2263 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:41459
I20260929 01:57:04.430606 2263 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:57:04.433804 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 2263
I20260929 01:57:04.434226 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 2117
I20260929 01:57:04.441463 2330 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:57:04.447538 2244 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:57:04.448149 2330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf: Bootstrap starting.
I20260929 01:57:04.477197 2330 log.cc:826] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf: Log is configured to *not* fsync() on all Append() calls
I20260929 01:57:04.494002 2121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87988700bf3342c7b8190f003fdd6524" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: false } } }
I20260929 01:57:04.494431 2121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: This master's current role is: LEADER
I20260929 01:57:04.505542 2330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:57:04.506718 2330 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf: Bootstrap complete.
I20260929 01:57:04.519477 2330 raft_consensus.cc:359] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: false } }
I20260929 01:57:04.521044 2330 raft_consensus.cc:740] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd0047e5034a4f1483071751913abfaf, State: Initialized, Role: FOLLOWER
I20260929 01:57:04.522178 2330 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: false } }
I20260929 01:57:04.523514 2340 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: false } } }
I20260929 01:57:04.524251 2340 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:04.525830 2330 sys_catalog.cc:565] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:57:04.528338 2340 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [sys.catalog]: SysCatalogTable state changed. Reason: New leader 87988700bf3342c7b8190f003fdd6524. Latest consensus state: current_term: 1 leader_uuid: "87988700bf3342c7b8190f003fdd6524" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: false } } }
I20260929 01:57:04.529065 2340 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:04.554234 2351 catalog_manager.cc:1308] Loaded cluster ID: 35495148a72749dd9d4814042afd810d
I20260929 01:57:04.556304 2351 catalog_manager.cc:1601] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf: loading cluster ID for follower catalog manager: success
I20260929 01:57:04.562053 2351 catalog_manager.cc:1623] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf: acquiring CA information for follower catalog manager: success
I20260929 01:57:04.572104 2351 catalog_manager.cc:1651] T 00000000000000000000000000000000 P fd0047e5034a4f1483071751913abfaf: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:57:04.584923 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 2117
I20260929 01:57:04.613046 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:41935
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:41459,127.0.71.125:38797,127.0.71.124:36653
--webserver_port=37423
--webserver_interface=127.0.71.65
--time_source=system_unsync
--tserver_master_addrs=127.0.71.126:41459,127.0.71.125:38797,127.0.71.124:36653,127.0.71.123:37063 with env {}
I20260929 01:57:04.997277 2121 sys_catalog.cc:455] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "87988700bf3342c7b8190f003fdd6524" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "fd0047e5034a4f1483071751913abfaf" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41459 } } peers { permanent_uuid: "c4adeb24a60248c587ab8d0dcecdde10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 38797 } } peers { permanent_uuid: "87988700bf3342c7b8190f003fdd6524" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36653 } } peers { permanent_uuid: "3405de609e5643159e835f3a17f44097" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 37063 } attrs { promote: false } } }
I20260929 01:57:04.997668 2121 sys_catalog.cc:458] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 [sys.catalog]: This master's current role is: LEADER
W20260929 01:57:05.077095 2357 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:57:05.077596 2357 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:57:05.077766 2357 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:57:05.088531 2357 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:57:05.088821 2357 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:57:05.101830 2357 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:41935
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=37423
--tserver_master_addrs=127.0.71.126:41459,127.0.71.125:38797,127.0.71.124:36653,127.0.71.123:37063
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:57:05.103978 2357 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:57:05.106611 2357 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:57:05.120121 2365 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:05.120389 2363 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:05.121029 2362 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:57:05.122172 2357 server_base.cc:1061] running on GCE node
I20260929 01:57:05.123071 2357 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:57:05.123297 2357 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:57:05.123415 2357 hybrid_clock.cc:648] HybridClock initialized: now 1790647025123392 us; error 0 us; skew 500 ppm
I20260929 01:57:05.123806 2357 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:57:05.126919 2357 webserver.cc:533] Webserver started at http://127.0.71.65:37423/ using document root <none> and password file <none>
I20260929 01:57:05.128114 2357 fs_manager.cc:362] Metadata directory not provided
I20260929 01:57:05.128288 2357 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:57:05.135851 2357 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.001s
I20260929 01:57:05.139860 2370 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:05.142230 2357 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.000s sys 0.002s
I20260929 01:57:05.142539 2357 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "09898dc208db41b8aacc96c92d4708f6"
format_stamp: "Formatted at 2026-09-29 01:57:01 on dist-test-slave-vz23"
I20260929 01:57:05.143550 2357 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:57:05.171639 2357 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:57:05.172819 2357 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:57:05.173221 2357 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:57:05.174448 2357 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:57:05.176839 2357 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:57:05.176949 2357 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:05.177101 2357 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:57:05.177181 2357 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:05.219017 2357 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:41935
I20260929 01:57:05.219058 2482 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:41935 every 8 connection(s)
I20260929 01:57:05.221019 2357 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:57:05.226855 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 2357
I20260929 01:57:05.275839 2483 heartbeater.cc:344] Connected to a master server at 127.0.71.124:36653
I20260929 01:57:05.276466 2483 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:05.278000 2483 heartbeater.cc:507] Master 127.0.71.124:36653 requested a full tablet report, sending...
I20260929 01:57:05.278385 2486 heartbeater.cc:344] Connected to a master server at 127.0.71.125:38797
I20260929 01:57:05.278617 2486 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:05.279402 2486 heartbeater.cc:507] Master 127.0.71.125:38797 requested a full tablet report, sending...
I20260929 01:57:05.280934 1694 ts_manager.cc:194] Re-registered known tserver with Master: 09898dc208db41b8aacc96c92d4708f6 (127.0.71.65:41935)
I20260929 01:57:05.281032 1623 ts_manager.cc:194] Re-registered known tserver with Master: 09898dc208db41b8aacc96c92d4708f6 (127.0.71.65:41935)
I20260929 01:57:05.282323 1694 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:44269
I20260929 01:57:05.282368 2484 heartbeater.cc:344] Connected to a master server at 127.0.71.123:37063
I20260929 01:57:05.282567 2484 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:05.283273 2484 heartbeater.cc:507] Master 127.0.71.123:37063 requested a full tablet report, sending...
I20260929 01:57:05.285643 2064 ts_manager.cc:194] Re-registered known tserver with Master: 09898dc208db41b8aacc96c92d4708f6 (127.0.71.65:41935)
I20260929 01:57:05.291795 2485 heartbeater.cc:344] Connected to a master server at 127.0.71.126:41459
I20260929 01:57:05.292068 2485 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:05.292656 2485 heartbeater.cc:507] Master 127.0.71.126:41459 requested a full tablet report, sending...
I20260929 01:57:05.295495 2295 ts_manager.cc:194] Registered new tserver with Master: 09898dc208db41b8aacc96c92d4708f6 (127.0.71.65:41935)
I20260929 01:57:05.303298 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 2357
I20260929 01:57:05.323235 2478 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:57:05.450646 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 2357
I20260929 01:57:05.470108 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:41935
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:41459,127.0.71.125:38797,127.0.71.124:36653
--webserver_port=37423
--webserver_interface=127.0.71.65
--time_source=system_unsync
--tserver_master_addrs=127.0.71.126:41459,127.0.71.125:38797,127.0.71.124:36653,127.0.71.123:37063
--fault_before_start=1.0 with env {}
W20260929 01:57:05.898870 2500 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:57:05.899264 2500 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:57:05.899351 2500 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:57:05.899425 2500 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:57:05.909060 2500 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:57:05.909291 2500 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:57:05.921373 2500 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:41935
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=37423
--tserver_master_addrs=127.0.71.126:41459,127.0.71.125:38797,127.0.71.124:36653,127.0.71.123:37063
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:57:05.923421 2500 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:57:05.925616 2500 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:57:05.938149 2505 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:05.938107 2506 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:05.939435 2508 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:57:05.940750 2500 server_base.cc:1061] running on GCE node
I20260929 01:57:05.941551 2500 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:57:05.941762 2500 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:57:05.941907 2500 hybrid_clock.cc:648] HybridClock initialized: now 1790647025941838 us; error 0 us; skew 500 ppm
I20260929 01:57:05.942276 2500 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:57:05.945186 2500 webserver.cc:533] Webserver started at http://127.0.71.65:37423/ using document root <none> and password file <none>
I20260929 01:57:05.946122 2500 fs_manager.cc:362] Metadata directory not provided
I20260929 01:57:05.946259 2500 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:57:05.953768 2500 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.001s
I20260929 01:57:05.957748 2513 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:05.959697 2500 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:57:05.960037 2500 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "09898dc208db41b8aacc96c92d4708f6"
format_stamp: "Formatted at 2026-09-29 01:57:01 on dist-test-slave-vz23"
I20260929 01:57:05.961212 2500 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:57:05.988888 2500 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:57:05.990097 2500 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:57:05.990473 2500 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:57:05.991696 2500 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:57:05.994055 2500 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:57:05.994194 2500 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:05.994343 2500 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:57:05.994453 2500 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:57:05.994932 2500 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:57:05.995098 2500 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:57:05.995195 2500 fault_injection.cc:59] injecting fault for kudu (pid 2500): FLAGS_fault_before_start (process will exit)
I20260929 01:57:06.015316 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:41935
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:41459,127.0.71.125:38797,127.0.71.124:36653
--webserver_port=37423
--webserver_interface=127.0.71.65
--time_source=system_unsync
--tserver_master_addrs=127.0.71.126:41459,127.0.71.125:38797,127.0.71.124:36653,127.0.71.123:37063 with env {}
W20260929 01:57:06.432400 2521 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:57:06.433020 2521 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:57:06.433189 2521 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:57:06.445073 2521 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:57:06.445333 2521 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:57:06.456554 2521 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:41935
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=37423
--tserver_master_addrs=127.0.71.126:41459,127.0.71.125:38797,127.0.71.124:36653,127.0.71.123:37063
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:57:06.458559 2521 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:57:06.460853 2521 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:57:06.474233 2529 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:06.474174 2526 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:06.474154 2527 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:57:06.476926 2521 server_base.cc:1061] running on GCE node
I20260929 01:57:06.477814 2521 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:57:06.478042 2521 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:57:06.478137 2521 hybrid_clock.cc:648] HybridClock initialized: now 1790647026478115 us; error 0 us; skew 500 ppm
I20260929 01:57:06.478585 2521 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:57:06.481791 2521 webserver.cc:533] Webserver started at http://127.0.71.65:37423/ using document root <none> and password file <none>
I20260929 01:57:06.482806 2521 fs_manager.cc:362] Metadata directory not provided
I20260929 01:57:06.482937 2521 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:57:06.491163 2521 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 01:57:06.495412 2534 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:06.497684 2521 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260929 01:57:06.497985 2521 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "09898dc208db41b8aacc96c92d4708f6"
format_stamp: "Formatted at 2026-09-29 01:57:01 on dist-test-slave-vz23"
I20260929 01:57:06.499039 2521 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:57:06.516037 2521 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:57:06.517493 2521 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:57:06.517882 2521 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:57:06.519129 2521 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:57:06.521670 2521 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:57:06.521804 2521 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:06.521970 2521 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:57:06.522043 2521 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:06.570546 2521 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:41935
I20260929 01:57:06.570613 2646 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:41935 every 8 connection(s)
I20260929 01:57:06.572772 2521 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:57:06.582468 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 2521
I20260929 01:57:06.582826 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 2521
I20260929 01:57:06.606629 2650 heartbeater.cc:344] Connected to a master server at 127.0.71.125:38797
I20260929 01:57:06.607259 2650 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:06.608857 2650 heartbeater.cc:507] Master 127.0.71.125:38797 requested a full tablet report, sending...
I20260929 01:57:06.609229 2647 heartbeater.cc:344] Connected to a master server at 127.0.71.124:36653
I20260929 01:57:06.609380 2647 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:06.610085 2647 heartbeater.cc:507] Master 127.0.71.124:36653 requested a full tablet report, sending...
I20260929 01:57:06.611292 2648 heartbeater.cc:344] Connected to a master server at 127.0.71.123:37063
I20260929 01:57:06.611512 2648 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:06.611457 1694 ts_manager.cc:194] Re-registered known tserver with Master: 09898dc208db41b8aacc96c92d4708f6 (127.0.71.65:41935)
I20260929 01:57:06.612048 1623 ts_manager.cc:194] Re-registered known tserver with Master: 09898dc208db41b8aacc96c92d4708f6 (127.0.71.65:41935)
I20260929 01:57:06.612215 2648 heartbeater.cc:507] Master 127.0.71.123:37063 requested a full tablet report, sending...
I20260929 01:57:06.612715 1694 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:52101
I20260929 01:57:06.613705 2064 ts_manager.cc:194] Re-registered known tserver with Master: 09898dc208db41b8aacc96c92d4708f6 (127.0.71.65:41935)
I20260929 01:57:06.619565 2649 heartbeater.cc:344] Connected to a master server at 127.0.71.126:41459
I20260929 01:57:06.619833 2649 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:06.620393 2649 heartbeater.cc:507] Master 127.0.71.126:41459 requested a full tablet report, sending...
I20260929 01:57:06.620666 2642 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:57:06.622577 2295 ts_manager.cc:194] Re-registered known tserver with Master: 09898dc208db41b8aacc96c92d4708f6 (127.0.71.65:41935)
I20260929 01:57:06.719271 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 2521
I20260929 01:57:06.739388 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 2263
I20260929 01:57:06.753095 2325 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:57:06.878825 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 2263
I20260929 01:57:06.902607 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 1590
I20260929 01:57:06.904167 1653 generic_service.cc:196] Checking for leaks (request via RPC)
W20260929 01:57:06.986526 1678 consensus_peers.cc:597] T 00000000000000000000000000000000 P 87988700bf3342c7b8190f003fdd6524 -> Peer fd0047e5034a4f1483071751913abfaf (127.0.71.126:41459): Couldn't send request to peer fd0047e5034a4f1483071751913abfaf. Status: Network error: Client connection negotiation failed: client connection to 127.0.71.126:41459: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 01:57:07.056164 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 1590
I20260929 01:57:07.080310 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 1660
I20260929 01:57:07.082271 1724 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:57:07.216115 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 1660
W20260929 01:57:07.240509 2050 connection.cc:573] server connection from 127.0.0.1:52890 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:57:07.240814 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 1926
I20260929 01:57:07.242266 2094 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:57:07.371511 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 1926
Shutting down hive metastore.
[ OK ] ExternalMiniClusterTest.TestAddMaster/3 (28922 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/4
2026-09-29T01:57:07Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:57:07Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/security.properties
2026-09-29 01:57:17: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260929 01:57:29.230468 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:36413
--webserver_interface=127.0.71.126
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:36413,127.0.71.125:46843,127.0.71.124:46285 with env {}
W20260929 01:57:29.638365 2953 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:57:29.638736 2953 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:57:29.638861 2953 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:57:29.648303 2953 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:57:29.648424 2953 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:57:29.648591 2953 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:57:29.648672 2953 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:57:29.660051 2953 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:36413,127.0.71.125:46843,127.0.71.124:46285
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:36413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:57:29.662030 2953 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:57:29.664628 2953 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:57:29.675738 2958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:29.676156 2961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:29.675798 2959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:57:29.677484 2953 server_base.cc:1061] running on GCE node
I20260929 01:57:29.678642 2953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:57:29.680696 2953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:57:29.682049 2953 hybrid_clock.cc:648] HybridClock initialized: now 1790647049682002 us; error 47 us; skew 500 ppm
I20260929 01:57:29.682621 2953 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:57:29.685882 2953 webserver.cc:533] Webserver started at http://127.0.71.126:40277/ using document root <none> and password file <none>
I20260929 01:57:29.686910 2953 fs_manager.cc:362] Metadata directory not provided
I20260929 01:57:29.687104 2953 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:57:29.687609 2953 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:57:29.690846 2953 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "5dd1d0fad1154df698b109127fc98e76"
format_stamp: "Formatted at 2026-09-29 01:57:29 on dist-test-slave-vz23"
I20260929 01:57:29.691956 2953 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "5dd1d0fad1154df698b109127fc98e76"
format_stamp: "Formatted at 2026-09-29 01:57:29 on dist-test-slave-vz23"
I20260929 01:57:29.699388 2953 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260929 01:57:29.704725 2967 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:29.706934 2953 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:57:29.707249 2953 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "5dd1d0fad1154df698b109127fc98e76"
format_stamp: "Formatted at 2026-09-29 01:57:29 on dist-test-slave-vz23"
I20260929 01:57:29.707587 2953 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:57:29.731194 2953 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:57:29.732297 2953 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:57:29.732692 2953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:57:29.758831 2953 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:36413
I20260929 01:57:29.758875 3018 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:36413 every 8 connection(s)
I20260929 01:57:29.760612 2953 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:57:29.767175 3019 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:57:29.768743 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 2953
I20260929 01:57:29.769033 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:57:29.770581 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:46843
--webserver_interface=127.0.71.125
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:36413,127.0.71.125:46843,127.0.71.124:46285 with env {}
I20260929 01:57:29.778332 3019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:57:29.801081 3019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:57:29.802693 2971 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.125:46843: connect: Connection refused (error 111)
W20260929 01:57:29.804905 3019 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:46843: Network error: Client connection negotiation failed: client connection to 127.0.71.125:46843: connect: Connection refused (error 111)
I20260929 01:57:29.854504 3019 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } attempt: 1
W20260929 01:57:29.857455 3019 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:46843: Network error: Client connection negotiation failed: client connection to 127.0.71.125:46843: connect: Connection refused (error 111)
I20260929 01:57:29.925901 3019 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } attempt: 2
W20260929 01:57:29.928720 3019 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:46843: Network error: Client connection negotiation failed: client connection to 127.0.71.125:46843: connect: Connection refused (error 111)
I20260929 01:57:30.020126 3019 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } attempt: 3
W20260929 01:57:30.022871 3019 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:46843: Network error: Client connection negotiation failed: client connection to 127.0.71.125:46843: connect: Connection refused (error 111)
I20260929 01:57:30.166296 3019 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } attempt: 4
W20260929 01:57:30.169332 3019 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:46843: Network error: Client connection negotiation failed: client connection to 127.0.71.125:46843: connect: Connection refused (error 111)
W20260929 01:57:30.182843 3021 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:57:30.183240 3021 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:57:30.183323 3021 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:57:30.192756 3021 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:57:30.192875 3021 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:57:30.192932 3021 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:57:30.192975 3021 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:57:30.204361 3021 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:36413,127.0.71.125:46843,127.0.71.124:46285
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:46843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:57:30.206238 3021 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:57:30.208719 3021 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:57:30.220191 3030 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:30.220281 3029 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:30.220836 3032 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:57:30.222128 3021 server_base.cc:1061] running on GCE node
I20260929 01:57:30.223100 3021 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:57:30.225191 3021 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:57:30.226526 3021 hybrid_clock.cc:648] HybridClock initialized: now 1790647050226423 us; error 91 us; skew 500 ppm
I20260929 01:57:30.227170 3021 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:57:30.230778 3021 webserver.cc:533] Webserver started at http://127.0.71.125:43269/ using document root <none> and password file <none>
I20260929 01:57:30.231839 3021 fs_manager.cc:362] Metadata directory not provided
I20260929 01:57:30.231968 3021 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:57:30.232594 3021 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:57:30.235688 3021 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-1/data/instance:
uuid: "9167d627ecab4454a801c180b4cb0bf9"
format_stamp: "Formatted at 2026-09-29 01:57:30 on dist-test-slave-vz23"
I20260929 01:57:30.236929 3021 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-1/wal/instance:
uuid: "9167d627ecab4454a801c180b4cb0bf9"
format_stamp: "Formatted at 2026-09-29 01:57:30 on dist-test-slave-vz23"
I20260929 01:57:30.244428 3021 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260929 01:57:30.249046 3038 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:30.251065 3021 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:57:30.251427 3021 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "9167d627ecab4454a801c180b4cb0bf9"
format_stamp: "Formatted at 2026-09-29 01:57:30 on dist-test-slave-vz23"
I20260929 01:57:30.251757 3021 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:57:30.275167 3021 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:57:30.276392 3021 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:57:30.276832 3021 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:57:30.302371 3021 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:46843
I20260929 01:57:30.302390 3089 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:46843 every 8 connection(s)
I20260929 01:57:30.304301 3021 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:57:30.309255 3090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:57:30.310465 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 3021
I20260929 01:57:30.310700 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:57:30.311877 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:46285
--webserver_interface=127.0.71.124
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:36413,127.0.71.125:46843,127.0.71.124:46285 with env {}
I20260929 01:57:30.316959 3090 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:57:30.335079 3090 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:57:30.348570 3090 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:57:30.350047 3042 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.124:46285: connect: Connection refused (error 111)
W20260929 01:57:30.352205 3090 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:46285: Network error: Client connection negotiation failed: client connection to 127.0.71.124:46285: connect: Connection refused (error 111)
I20260929 01:57:30.401611 3090 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } attempt: 1
W20260929 01:57:30.404320 3090 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:46285: Network error: Client connection negotiation failed: client connection to 127.0.71.124:46285: connect: Connection refused (error 111)
I20260929 01:57:30.468719 3019 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } attempt: 5
I20260929 01:57:30.472676 3090 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } attempt: 2
W20260929 01:57:30.475564 3090 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:46285: Network error: Client connection negotiation failed: client connection to 127.0.71.124:46285: connect: Connection refused (error 111)
I20260929 01:57:30.477451 3019 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:57:30.479974 3019 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:46285: Network error: Client connection negotiation failed: client connection to 127.0.71.124:46285: connect: Connection refused (error 111)
I20260929 01:57:30.531338 3019 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } attempt: 1
W20260929 01:57:30.533790 3019 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:46285: Network error: Client connection negotiation failed: client connection to 127.0.71.124:46285: connect: Connection refused (error 111)
I20260929 01:57:30.566923 3090 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } attempt: 3
W20260929 01:57:30.569624 3090 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:46285: Network error: Client connection negotiation failed: client connection to 127.0.71.124:46285: connect: Connection refused (error 111)
I20260929 01:57:30.602139 3019 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } attempt: 2
W20260929 01:57:30.604708 3019 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:46285: Network error: Client connection negotiation failed: client connection to 127.0.71.124:46285: connect: Connection refused (error 111)
I20260929 01:57:30.711027 3019 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } attempt: 3
I20260929 01:57:30.712911 3090 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } attempt: 4
W20260929 01:57:30.713922 3019 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:46285: Network error: Client connection negotiation failed: client connection to 127.0.71.124:46285: connect: Connection refused (error 111)
W20260929 01:57:30.715123 3090 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:46285: Network error: Client connection negotiation failed: client connection to 127.0.71.124:46285: connect: Connection refused (error 111)
W20260929 01:57:30.720412 3092 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:57:30.720927 3092 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:57:30.721004 3092 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:57:30.730899 3092 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:57:30.731050 3092 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:57:30.731114 3092 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:57:30.731156 3092 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:57:30.742846 3092 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:36413,127.0.71.125:46843,127.0.71.124:46285
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:46285
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:57:30.745011 3092 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:57:30.747246 3092 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:57:30.759470 3101 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:30.760114 3104 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:30.759611 3102 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:57:30.762634 3092 server_base.cc:1061] running on GCE node
I20260929 01:57:30.763695 3092 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:57:30.765676 3092 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:57:30.767113 3092 hybrid_clock.cc:648] HybridClock initialized: now 1790647050767013 us; error 98 us; skew 500 ppm
I20260929 01:57:30.767629 3092 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:57:30.770640 3092 webserver.cc:533] Webserver started at http://127.0.71.124:35259/ using document root <none> and password file <none>
I20260929 01:57:30.771620 3092 fs_manager.cc:362] Metadata directory not provided
I20260929 01:57:30.772003 3092 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:57:30.772630 3092 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:57:30.775758 3092 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-2/data/instance:
uuid: "126e333b2eab482c8c57e9e4938169bd"
format_stamp: "Formatted at 2026-09-29 01:57:30 on dist-test-slave-vz23"
I20260929 01:57:30.776983 3092 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-2/wal/instance:
uuid: "126e333b2eab482c8c57e9e4938169bd"
format_stamp: "Formatted at 2026-09-29 01:57:30 on dist-test-slave-vz23"
I20260929 01:57:30.784246 3092 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.002s sys 0.004s
I20260929 01:57:30.789116 3110 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:30.791172 3092 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:57:30.791473 3092 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "126e333b2eab482c8c57e9e4938169bd"
format_stamp: "Formatted at 2026-09-29 01:57:30 on dist-test-slave-vz23"
I20260929 01:57:30.791785 3092 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:57:30.835801 3092 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:57:30.837066 3092 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:57:30.837445 3092 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:57:30.868305 3161 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:46285 every 8 connection(s)
I20260929 01:57:30.868325 3092 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:46285
I20260929 01:57:30.871079 3092 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:57:30.879442 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 3092
I20260929 01:57:30.879742 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:57:30.879761 3162 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:57:30.890172 3162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:57:30.891278 3019 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } attempt: 4
I20260929 01:57:30.907837 3162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:57:30.917524 3019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76: Bootstrap starting.
I20260929 01:57:30.920238 3162 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:57:30.922983 3019 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76: Neither blocks nor log segments found. Creating new log.
I20260929 01:57:30.925062 3019 log.cc:826] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76: Log is configured to *not* fsync() on all Append() calls
I20260929 01:57:30.930333 3019 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76: No bootstrap required, opened a new log
I20260929 01:57:30.935997 3162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd: Bootstrap starting.
I20260929 01:57:30.939966 3019 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } }
I20260929 01:57:30.940485 3019 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:57:30.940629 3019 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd1d0fad1154df698b109127fc98e76, State: Initialized, Role: FOLLOWER
I20260929 01:57:30.941556 3019 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } }
I20260929 01:57:30.941891 3162 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd: Neither blocks nor log segments found. Creating new log.
I20260929 01:57:30.942942 3170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } }
I20260929 01:57:30.943508 3170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:30.943567 3162 log.cc:826] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd: Log is configured to *not* fsync() on all Append() calls
I20260929 01:57:30.944707 3019 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:57:30.948545 3162 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd: No bootstrap required, opened a new log
I20260929 01:57:30.958665 3162 raft_consensus.cc:359] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } }
I20260929 01:57:30.959220 3162 raft_consensus.cc:385] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:57:30.959410 3162 raft_consensus.cc:740] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 126e333b2eab482c8c57e9e4938169bd, State: Initialized, Role: FOLLOWER
I20260929 01:57:30.960616 3162 consensus_queue.cc:260] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } }
I20260929 01:57:30.963327 3184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } }
I20260929 01:57:30.963891 3184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:57:30.964926 3182 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:57:30.965188 3182 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:57:30.965342 3162 sys_catalog.cc:565] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:57:30.960028 2953 master_runner.cc:187] Error getting master registration for 127.0.71.125:46843: OK, instance_id { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" instance_seqno: 1790647050296212 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:57:30.968600 2953 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:57:30.981884 3195 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:57:30.982316 3195 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 01:57:30.987072 2973 tablet.cc:2433] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:57:30.984779 3092 master_runner.cc:187] Error getting master registration for 127.0.71.125:46843: OK, instance_id { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" instance_seqno: 1790647050296212 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:57:30.994374 3092 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:57:31.014483 3090 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } attempt: 5
I20260929 01:57:31.035419 3170 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:57:31.035784 3170 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } }
I20260929 01:57:31.038175 3170 leader_election.cc:290] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 9167d627ecab4454a801c180b4cb0bf9 (127.0.71.125:46843), 126e333b2eab482c8c57e9e4938169bd (127.0.71.124:46285)
I20260929 01:57:31.039697 3137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5dd1d0fad1154df698b109127fc98e76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "126e333b2eab482c8c57e9e4938169bd" is_pre_election: true
I20260929 01:57:31.040426 3137 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 5dd1d0fad1154df698b109127fc98e76 in term 0.
I20260929 01:57:31.041416 2971 leader_election.cc:304] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 126e333b2eab482c8c57e9e4938169bd, 5dd1d0fad1154df698b109127fc98e76; no voters:
I20260929 01:57:31.042166 3170 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:57:31.042366 3170 raft_consensus.cc:493] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:57:31.042495 3170 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:57:31.043497 3090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9: Bootstrap starting.
I20260929 01:57:31.046664 3170 raft_consensus.cc:515] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } }
I20260929 01:57:31.049292 3170 leader_election.cc:290] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [CANDIDATE]: Term 1 election: Requested vote from peers 9167d627ecab4454a801c180b4cb0bf9 (127.0.71.125:46843), 126e333b2eab482c8c57e9e4938169bd (127.0.71.124:46285)
I20260929 01:57:31.049475 3137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5dd1d0fad1154df698b109127fc98e76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "126e333b2eab482c8c57e9e4938169bd"
I20260929 01:57:31.049846 3137 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:57:31.054282 3137 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd1d0fad1154df698b109127fc98e76 in term 1.
I20260929 01:57:31.055256 2971 leader_election.cc:304] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 126e333b2eab482c8c57e9e4938169bd, 5dd1d0fad1154df698b109127fc98e76; no voters:
I20260929 01:57:31.056005 3170 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:57:31.064329 3090 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9: Neither blocks nor log segments found. Creating new log.
I20260929 01:57:31.067826 3090 log.cc:826] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9: Log is configured to *not* fsync() on all Append() calls
I20260929 01:57:31.070596 3170 raft_consensus.cc:697] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [term 1 LEADER]: Becoming Leader. State: Replica: 5dd1d0fad1154df698b109127fc98e76, State: Running, Role: LEADER
I20260929 01:57:31.072180 3170 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } }
I20260929 01:57:31.081560 3090 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9: No bootstrap required, opened a new log
W20260929 01:57:31.090152 3116 tablet.cc:2433] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:57:31.097769 3197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dd1d0fad1154df698b109127fc98e76. Latest consensus state: current_term: 1 leader_uuid: "5dd1d0fad1154df698b109127fc98e76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } }
I20260929 01:57:31.098229 3197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [sys.catalog]: This master's current role is: LEADER
I20260929 01:57:31.105407 3090 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } }
I20260929 01:57:31.105902 3090 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:57:31.106065 3090 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9167d627ecab4454a801c180b4cb0bf9, State: Initialized, Role: FOLLOWER
I20260929 01:57:31.107028 3090 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } }
I20260929 01:57:31.109193 3201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } }
I20260929 01:57:31.110330 3201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:31.116219 3200 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:57:31.119230 3200 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:57:31.123106 3064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5dd1d0fad1154df698b109127fc98e76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9167d627ecab4454a801c180b4cb0bf9"
I20260929 01:57:31.123871 3064 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:57:31.124759 3065 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "5dd1d0fad1154df698b109127fc98e76" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "9167d627ecab4454a801c180b4cb0bf9" is_pre_election: true
I20260929 01:57:31.131134 3064 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 5dd1d0fad1154df698b109127fc98e76 in term 1.
I20260929 01:57:31.123095 3090 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:57:31.145058 3137 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [term 1 FOLLOWER]: Refusing update from remote peer 5dd1d0fad1154df698b109127fc98e76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:57:31.146657 3197 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:57:31.160347 3064 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [term 1 FOLLOWER]: Refusing update from remote peer 5dd1d0fad1154df698b109127fc98e76: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:57:31.161810 3197 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:57:31.186301 3201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dd1d0fad1154df698b109127fc98e76. Latest consensus state: current_term: 1 leader_uuid: "5dd1d0fad1154df698b109127fc98e76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } }
I20260929 01:57:31.186625 3201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:31.195176 3202 mvcc.cc:204] Tried to move back new op lower bound from 7334490321473613824 to 7334490321206824960. Current Snapshot: MvccSnapshot[applied={T|T < 7334490321473613824}]
I20260929 01:57:31.202884 3184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dd1d0fad1154df698b109127fc98e76. Latest consensus state: current_term: 1 leader_uuid: "5dd1d0fad1154df698b109127fc98e76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } }
I20260929 01:57:31.203194 3184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:31.212188 3201 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5dd1d0fad1154df698b109127fc98e76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } }
I20260929 01:57:31.212491 3201 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:31.213634 3197 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5dd1d0fad1154df698b109127fc98e76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } }
I20260929 01:57:31.213927 3197 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [sys.catalog]: This master's current role is: LEADER
I20260929 01:57:31.214852 3170 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5dd1d0fad1154df698b109127fc98e76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } }
I20260929 01:57:31.215142 3170 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [sys.catalog]: This master's current role is: LEADER
I20260929 01:57:31.216353 3200 catalog_manager.cc:1396] Generated new cluster ID: 4e17683b73c2438f9fcbc54fde2e73dd
I20260929 01:57:31.216499 3200 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:57:31.216408 3184 sys_catalog.cc:455] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5dd1d0fad1154df698b109127fc98e76" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } }
I20260929 01:57:31.216679 3184 sys_catalog.cc:458] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:31.221690 3206 mvcc.cc:204] Tried to move back new op lower bound from 7334490321473613824 to 7334490321206824960. Current Snapshot: MvccSnapshot[applied={T|T < 7334490321473613824 or (T in {7334490321473613824})}]
I20260929 01:57:31.250167 3200 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 01:57:31.252856 3200 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:57:31.276926 3200 catalog_manager.cc:6219] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76: Generated new TSK 0
I20260929 01:57:31.279063 3200 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:57:31.274243 3221 catalog_manager.cc:818] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 01:57:31.287248 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:36413,127.0.71.125:46843,127.0.71.124:46285
--builtin_ntp_servers=127.0.71.84:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
W20260929 01:57:31.759601 3222 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:57:31.760607 3222 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:57:31.761176 3222 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:57:31.776960 3222 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:57:31.777264 3222 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:57:31.792721 3222 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:36413,127.0.71.125:46843,127.0.71.124:46285
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:57:31.794756 3222 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:57:31.796908 3222 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:57:31.808974 3230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:31.809012 3227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:31.809072 3228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:57:31.810199 3222 server_base.cc:1061] running on GCE node
I20260929 01:57:31.811002 3222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:57:31.812254 3222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:57:31.813519 3222 hybrid_clock.cc:648] HybridClock initialized: now 1790647051813477 us; error 48 us; skew 500 ppm
I20260929 01:57:31.813983 3222 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:57:31.817140 3222 webserver.cc:533] Webserver started at http://127.0.71.65:40881/ using document root <none> and password file <none>
I20260929 01:57:31.818150 3222 fs_manager.cc:362] Metadata directory not provided
I20260929 01:57:31.818306 3222 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:57:31.818732 3222 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:57:31.821817 3222 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "1dc520dc7c1b4d46a8409d4577afcc0c"
format_stamp: "Formatted at 2026-09-29 01:57:31 on dist-test-slave-vz23"
I20260929 01:57:31.823051 3222 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "1dc520dc7c1b4d46a8409d4577afcc0c"
format_stamp: "Formatted at 2026-09-29 01:57:31 on dist-test-slave-vz23"
I20260929 01:57:31.830683 3222 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260929 01:57:31.835752 3236 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:31.838510 3222 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260929 01:57:31.838850 3222 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "1dc520dc7c1b4d46a8409d4577afcc0c"
format_stamp: "Formatted at 2026-09-29 01:57:31 on dist-test-slave-vz23"
I20260929 01:57:31.839149 3222 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:57:31.881649 3222 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:57:31.882797 3222 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:57:31.883162 3222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:57:31.884406 3222 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:57:31.886996 3222 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:57:31.887168 3222 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:31.887344 3222 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:57:31.887447 3222 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:31.928941 3222 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:39481
I20260929 01:57:31.929013 3348 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:39481 every 8 connection(s)
I20260929 01:57:31.930656 3222 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:57:31.938795 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 3222
I20260929 01:57:31.939051 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
I20260929 01:57:31.954846 3350 heartbeater.cc:344] Connected to a master server at 127.0.71.126:36413
I20260929 01:57:31.955503 3350 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:31.955888 3351 heartbeater.cc:344] Connected to a master server at 127.0.71.125:46843
I20260929 01:57:31.956050 3351 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:31.957031 3350 heartbeater.cc:507] Master 127.0.71.126:36413 requested a full tablet report, sending...
I20260929 01:57:31.957055 3351 heartbeater.cc:507] Master 127.0.71.125:46843 requested a full tablet report, sending...
I20260929 01:57:31.957909 3349 heartbeater.cc:344] Connected to a master server at 127.0.71.124:46285
I20260929 01:57:31.958534 3349 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:31.959270 3349 heartbeater.cc:507] Master 127.0.71.124:46285 requested a full tablet report, sending...
I20260929 01:57:31.960054 2984 ts_manager.cc:194] Registered new tserver with Master: 1dc520dc7c1b4d46a8409d4577afcc0c (127.0.71.65:39481)
I20260929 01:57:31.960268 3055 ts_manager.cc:194] Registered new tserver with Master: 1dc520dc7c1b4d46a8409d4577afcc0c (127.0.71.65:39481)
I20260929 01:57:31.961844 3127 ts_manager.cc:194] Registered new tserver with Master: 1dc520dc7c1b4d46a8409d4577afcc0c (127.0.71.65:39481)
I20260929 01:57:31.962620 2984 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:50695
I20260929 01:57:31.973749 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:57:31.976221 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.123:38679
--webserver_interface=127.0.71.123
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:36413,127.0.71.125:46843,127.0.71.124:46285,127.0.71.123:38679 with env {}
I20260929 01:57:31.984938 3195 catalog_manager.cc:1308] Loaded cluster ID: 4e17683b73c2438f9fcbc54fde2e73dd
I20260929 01:57:31.985097 3195 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd: loading cluster ID for follower catalog manager: success
I20260929 01:57:31.989864 3195 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd: acquiring CA information for follower catalog manager: success
I20260929 01:57:31.993047 3195 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:57:32.289577 3221 catalog_manager.cc:1308] Loaded cluster ID: 4e17683b73c2438f9fcbc54fde2e73dd
I20260929 01:57:32.289748 3221 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9: loading cluster ID for follower catalog manager: success
I20260929 01:57:32.295122 3221 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9: acquiring CA information for follower catalog manager: success
I20260929 01:57:32.299142 3221 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 01:57:32.404260 3359 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:57:32.404853 3359 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:57:32.405005 3359 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:57:32.415848 3359 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:57:32.415985 3359 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:57:32.416097 3359 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:57:32.416201 3359 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:57:32.428979 3359 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:36413,127.0.71.125:46843,127.0.71.124:46285,127.0.71.123:38679
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.123:38679
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.0.71.123
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:57:32.430965 3359 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:57:32.433184 3359 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:57:32.444536 3365 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:32.444993 3366 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:32.445914 3368 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:57:32.448081 3359 server_base.cc:1061] running on GCE node
I20260929 01:57:32.449429 3359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:57:32.451647 3359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:57:32.452989 3359 hybrid_clock.cc:648] HybridClock initialized: now 1790647052452901 us; error 75 us; skew 500 ppm
I20260929 01:57:32.453464 3359 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:57:32.456524 3359 webserver.cc:533] Webserver started at http://127.0.71.123:44675/ using document root <none> and password file <none>
I20260929 01:57:32.457504 3359 fs_manager.cc:362] Metadata directory not provided
I20260929 01:57:32.457700 3359 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:57:32.458330 3359 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:57:32.461659 3359 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-3/data/instance:
uuid: "742125c6ce0045198bfe0fe0331cb565"
format_stamp: "Formatted at 2026-09-29 01:57:32 on dist-test-slave-vz23"
I20260929 01:57:32.462468 3359 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-3/wal/instance:
uuid: "742125c6ce0045198bfe0fe0331cb565"
format_stamp: "Formatted at 2026-09-29 01:57:32 on dist-test-slave-vz23"
I20260929 01:57:32.469323 3359 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260929 01:57:32.474159 3374 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:32.476055 3359 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.003s
I20260929 01:57:32.476293 3359 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "742125c6ce0045198bfe0fe0331cb565"
format_stamp: "Formatted at 2026-09-29 01:57:32 on dist-test-slave-vz23"
I20260929 01:57:32.476619 3359 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:57:32.498541 3359 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:57:32.499917 3359 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:57:32.500371 3359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:57:32.531234 3359 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.123:38679
I20260929 01:57:32.531227 3425 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.123:38679 every 8 connection(s)
I20260929 01:57:32.533763 3359 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
I20260929 01:57:32.539849 3426 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:57:32.543679 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 3359
I20260929 01:57:32.551551 3426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:57:32.579694 3426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:57:32.590152 3426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:57:32.599601 3426 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.123" port: 38679 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:57:32.615268 3426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565: Bootstrap starting.
I20260929 01:57:32.620780 3426 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565: Neither blocks nor log segments found. Creating new log.
I20260929 01:57:32.622646 3426 log.cc:826] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565: Log is configured to *not* fsync() on all Append() calls
I20260929 01:57:32.627097 3426 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565: No bootstrap required, opened a new log
I20260929 01:57:32.636487 3426 raft_consensus.cc:359] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 38679 } }
I20260929 01:57:32.637094 3426 raft_consensus.cc:385] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:57:32.637252 3426 raft_consensus.cc:740] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 742125c6ce0045198bfe0fe0331cb565, State: Initialized, Role: FOLLOWER
I20260929 01:57:32.638351 3426 consensus_queue.cc:260] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 38679 } }
I20260929 01:57:32.639591 3435 sys_catalog.cc:455] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 38679 } } }
I20260929 01:57:32.639995 3435 sys_catalog.cc:458] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:32.641726 3426 sys_catalog.cc:565] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:57:32.649226 3446 catalog_manager.cc:818] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
I20260929 01:57:32.667588 3359 master_runner.cc:429] Detected that this master 742125c6ce0045198bfe0fe0331cb565 is joining an existing cluster
I20260929 01:57:32.667781 3359 catalog_manager.cc:1037] Shutting down the Catalog Manager
I20260929 01:57:32.668648 3359 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260929 01:57:32.669198 3359 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260929 01:57:32.669492 3359 tablet_replica.cc:333] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565: stopping tablet replica
I20260929 01:57:32.672362 3359 master_runner.cc:434] Initiating AddMaster RPC to add 127.0.71.123:38679
I20260929 01:57:32.712860 2984 catalog_manager.cc:7351] Initiating ChangeConfig request to add master 127.0.71.123:38679: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "742125c6ce0045198bfe0fe0331cb565"
member_type: NON_VOTER
last_known_addr {
host: "127.0.71.123"
port: 38679
}
attrs {
promote: true
}
}
dest_uuid: "5dd1d0fad1154df698b109127fc98e76"
cas_config_opid_index: -1
I20260929 01:57:32.714762 2984 consensus_queue.cc:237] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } }
I20260929 01:57:32.719473 3064 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [term 1 FOLLOWER]: Refusing update from remote peer 5dd1d0fad1154df698b109127fc98e76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20260929 01:57:32.720252 2971 consensus_peers.cc:597] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 -> Peer 742125c6ce0045198bfe0fe0331cb565 (127.0.71.123:38679): Couldn't send request to peer 742125c6ce0045198bfe0fe0331cb565. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260929 01:57:32.721057 3457 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:57:32.721426 3137 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [term 1 FOLLOWER]: Refusing update from remote peer 5dd1d0fad1154df698b109127fc98e76: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 01:57:32.722638 3457 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [LEADER]: Connected to new peer: Peer: permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:57:32.727181 3217 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 742125c6ce0045198bfe0fe0331cb565 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } } }
I20260929 01:57:32.728716 3064 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 742125c6ce0045198bfe0fe0331cb565 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } } }
I20260929 01:57:32.728722 3137 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 742125c6ce0045198bfe0fe0331cb565 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } } }
I20260929 01:57:32.733357 3217 catalog_manager.cc:7355] Successfully completed master ChangeConfig request to add master 127.0.71.123:38679
I20260929 01:57:32.734685 3359 master.cc:562] Master@127.0.71.123:38679 shutting down...
I20260929 01:57:32.753909 3460 sys_catalog.cc:455] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5dd1d0fad1154df698b109127fc98e76" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } } }
I20260929 01:57:32.755512 3460 sys_catalog.cc:458] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:57:32.787006 2673 connection.cc:573] client connection to 127.0.71.123:38679 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:57:32.790731 3359 master.cc:584] Master@127.0.71.123:38679 shutdown complete.
I20260929 01:57:32.790931 3359 master_runner.cc:306] Clearing existing system tablet
I20260929 01:57:32.791414 3359 fs_manager.cc:362] Metadata directory not provided
I20260929 01:57:32.791585 3359 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
W20260929 01:57:32.795251 2971 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.71.124:46285, user_credentials={real_user=slave}} blocked reactor thread for 51952.8us
I20260929 01:57:32.804780 3359 fs_manager.cc:714] Time spent opening directory manager: real 0.012s user 0.002s sys 0.000s
I20260929 01:57:32.810693 3465 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:32.812076 3359 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:57:32.812866 3359 fs_manager.cc:730] Time spent opening block manager: real 0.007s user 0.004s sys 0.000s
I20260929 01:57:32.813076 3359 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "742125c6ce0045198bfe0fe0331cb565"
format_stamp: "Formatted at 2026-09-29 01:57:32 on dist-test-slave-vz23"
I20260929 01:57:32.813086 3461 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "5dd1d0fad1154df698b109127fc98e76" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } } }
I20260929 01:57:32.813598 3461 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:32.814932 3217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5dd1d0fad1154df698b109127fc98e76" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } } }
I20260929 01:57:32.814932 3457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "5dd1d0fad1154df698b109127fc98e76" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } } }
I20260929 01:57:32.815290 3217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [sys.catalog]: This master's current role is: LEADER
I20260929 01:57:32.815310 3457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [sys.catalog]: This master's current role is: LEADER
I20260929 01:57:32.815882 3457 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5dd1d0fad1154df698b109127fc98e76" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } } }
I20260929 01:57:32.816139 3457 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [sys.catalog]: This master's current role is: LEADER
I20260929 01:57:32.818755 3359 ts_tablet_manager.cc:1962] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:57:32.824560 3359 ts_tablet_manager.cc:1975] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 01:57:32.824802 3359 log.cc:1199] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260929 01:57:32.825292 3359 ts_tablet_manager.cc:1996] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565: Deleting consensus metadata
I20260929 01:57:32.825567 3359 master_runner.cc:316] Copying system tablet from 127.0.71.126:36413
I20260929 01:57:32.828856 3359 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565: tablet copy: Beginning tablet copy session from remote peer at address 127.0.71.126:36413
I20260929 01:57:32.837477 3004 tablet_copy_service.cc:140] P 5dd1d0fad1154df698b109127fc98e76: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 742125c6ce0045198bfe0fe0331cb565 ({username='slave'} at 127.0.0.1:45728)
I20260929 01:57:32.837832 3004 tablet_copy_service.cc:161] P 5dd1d0fad1154df698b109127fc98e76: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 742125c6ce0045198bfe0fe0331cb565 at {username='slave'} at 127.0.0.1:45728: session id = 742125c6ce0045198bfe0fe0331cb565-00000000000000000000000000000000
I20260929 01:57:32.842505 3004 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 01:57:32.846082 3359 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 01:57:32.851740 3359 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565: tablet copy: Starting download of 0 data blocks...
I20260929 01:57:32.852090 3359 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565: tablet copy: Starting download of 1 WAL segments...
I20260929 01:57:32.855619 3359 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 01:57:32.860230 3004 tablet_copy_service.cc:342] P 5dd1d0fad1154df698b109127fc98e76: Request end of tablet copy session 742125c6ce0045198bfe0fe0331cb565-00000000000000000000000000000000 received from {username='slave'} at 127.0.0.1:45728
I20260929 01:57:32.860585 3004 tablet_copy_service.cc:434] P 5dd1d0fad1154df698b109127fc98e76: ending tablet copy session 742125c6ce0045198bfe0fe0331cb565-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 742125c6ce0045198bfe0fe0331cb565
W20260929 01:57:32.868785 3359 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260929 01:57:32.873112 3359 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:57:32.876225 3474 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:32.876317 3473 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:57:32.882704 3359 server_base.cc:1061] running on GCE node
W20260929 01:57:32.882694 3476 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:57:32.883385 3359 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:57:32.884128 3359 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:57:32.885402 3359 hybrid_clock.cc:648] HybridClock initialized: now 1790647052885319 us; error 62 us; skew 500 ppm
I20260929 01:57:32.885677 3359 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:57:32.887535 3359 webserver.cc:533] Webserver started at http://127.0.71.123:45265/ using document root <none> and password file <none>
I20260929 01:57:32.887923 3359 fs_manager.cc:362] Metadata directory not provided
I20260929 01:57:32.888054 3359 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:57:32.891224 3359 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:57:32.893111 3482 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:32.893747 3359 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:57:32.893957 3359 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "742125c6ce0045198bfe0fe0331cb565"
format_stamp: "Formatted at 2026-09-29 01:57:32 on dist-test-slave-vz23"
I20260929 01:57:32.894279 3359 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:57:32.947319 3359 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:57:32.948103 3359 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:57:32.966545 3350 heartbeater.cc:499] Master 127.0.71.126:36413 was elected leader, sending a full tablet report...
I20260929 01:57:32.969070 3359 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.123:38679
I20260929 01:57:32.969147 3533 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.123:38679 every 8 connection(s)
I20260929 01:57:32.970156 3359 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
I20260929 01:57:32.974505 3534 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:57:32.977025 3534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565: Bootstrap starting.
I20260929 01:57:32.992297 3534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:57:32.993121 3534 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565: Bootstrap complete.
I20260929 01:57:32.994899 3534 raft_consensus.cc:359] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } }
I20260929 01:57:32.995199 3534 raft_consensus.cc:740] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 742125c6ce0045198bfe0fe0331cb565, State: Initialized, Role: LEARNER
I20260929 01:57:32.995527 3534 consensus_queue.cc:260] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } }
I20260929 01:57:32.996937 3534 sys_catalog.cc:565] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:57:32.996743 3538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } } }
I20260929 01:57:32.997128 3538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565 [sys.catalog]: This master's current role is: LEARNER
I20260929 01:57:33.007831 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 3222
I20260929 01:57:33.011825 3549 catalog_manager.cc:1308] Loaded cluster ID: 4e17683b73c2438f9fcbc54fde2e73dd
I20260929 01:57:33.011960 3549 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565: loading cluster ID for follower catalog manager: success
I20260929 01:57:33.016623 3549 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565: acquiring CA information for follower catalog manager: success
I20260929 01:57:33.020241 3549 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:57:33.023861 3344 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:57:33.158458 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 3222
I20260929 01:57:33.178552 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:39481
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:36413,127.0.71.125:46843,127.0.71.124:46285
--webserver_port=40881
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:36413,127.0.71.125:46843,127.0.71.124:46285,127.0.71.123:38679 with env {}
I20260929 01:57:33.331547 3509 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260929 01:57:33.332772 3538 sys_catalog.cc:455] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 5dd1d0fad1154df698b109127fc98e76. Latest consensus state: current_term: 1 leader_uuid: "5dd1d0fad1154df698b109127fc98e76" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } } }
I20260929 01:57:33.333069 3538 sys_catalog.cc:458] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565 [sys.catalog]: This master's current role is: LEARNER
W20260929 01:57:33.625511 3553 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:57:33.625990 3553 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:57:33.626150 3553 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:57:33.637310 3553 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:57:33.637624 3553 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:57:33.649564 3553 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:39481
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=40881
--tserver_master_addrs=127.0.71.126:36413,127.0.71.125:46843,127.0.71.124:46285,127.0.71.123:38679
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:57:33.651626 3553 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:57:33.654280 3553 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:57:33.668908 3560 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:33.669070 3559 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:33.671726 3562 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:57:33.673839 3553 server_base.cc:1061] running on GCE node
I20260929 01:57:33.674742 3553 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:57:33.676251 3553 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:57:33.679850 3553 hybrid_clock.cc:648] HybridClock initialized: now 1790647053679730 us; error 75 us; skew 500 ppm
I20260929 01:57:33.680619 3553 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:57:33.683768 3553 webserver.cc:533] Webserver started at http://127.0.71.65:40881/ using document root <none> and password file <none>
I20260929 01:57:33.684861 3553 fs_manager.cc:362] Metadata directory not provided
I20260929 01:57:33.685051 3553 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:57:33.692960 3553 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 01:57:33.697749 3568 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:33.700222 3553 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260929 01:57:33.700567 3553 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "1dc520dc7c1b4d46a8409d4577afcc0c"
format_stamp: "Formatted at 2026-09-29 01:57:31 on dist-test-slave-vz23"
I20260929 01:57:33.701785 3553 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:57:33.726102 3553 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:57:33.727147 3553 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:57:33.727509 3553 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:57:33.728782 3553 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:57:33.731045 3553 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:57:33.731148 3553 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:33.731257 3553 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:57:33.731305 3553 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:33.778609 3553 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:39481
I20260929 01:57:33.778638 3680 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:39481 every 8 connection(s)
I20260929 01:57:33.780841 3553 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:57:33.784934 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 3553
I20260929 01:57:33.807538 3682 heartbeater.cc:344] Connected to a master server at 127.0.71.124:46285
I20260929 01:57:33.808092 3682 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:33.809501 3681 heartbeater.cc:344] Connected to a master server at 127.0.71.123:38679
I20260929 01:57:33.809523 3682 heartbeater.cc:507] Master 127.0.71.124:46285 requested a full tablet report, sending...
I20260929 01:57:33.809718 3681 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:33.810420 3681 heartbeater.cc:507] Master 127.0.71.123:38679 requested a full tablet report, sending...
I20260929 01:57:33.810464 3683 heartbeater.cc:344] Connected to a master server at 127.0.71.126:36413
I20260929 01:57:33.810612 3683 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:33.811209 3683 heartbeater.cc:507] Master 127.0.71.126:36413 requested a full tablet report, sending...
I20260929 01:57:33.812968 3127 ts_manager.cc:194] Re-registered known tserver with Master: 1dc520dc7c1b4d46a8409d4577afcc0c (127.0.71.65:39481)
I20260929 01:57:33.813459 3499 ts_manager.cc:194] Registered new tserver with Master: 1dc520dc7c1b4d46a8409d4577afcc0c (127.0.71.65:39481)
I20260929 01:57:33.814759 2984 ts_manager.cc:194] Re-registered known tserver with Master: 1dc520dc7c1b4d46a8409d4577afcc0c (127.0.71.65:39481)
I20260929 01:57:33.815842 2984 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:50165
I20260929 01:57:33.817211 3684 heartbeater.cc:344] Connected to a master server at 127.0.71.125:46843
I20260929 01:57:33.817456 3684 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:33.818405 3684 heartbeater.cc:507] Master 127.0.71.125:46843 requested a full tablet report, sending...
I20260929 01:57:33.819891 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:57:33.820290 3055 ts_manager.cc:194] Re-registered known tserver with Master: 1dc520dc7c1b4d46a8409d4577afcc0c (127.0.71.65:39481)
I20260929 01:57:33.833179 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:57:33.833402 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 2953
I20260929 01:57:33.834610 3014 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:57:33.967772 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 2953
I20260929 01:57:33.967643 3217 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "5dd1d0fad1154df698b109127fc98e76" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } } }
I20260929 01:57:33.967978 3217 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [sys.catalog]: This master's current role is: LEADER
W20260929 01:57:33.991382 3485 connection.cc:573] server connection from 127.0.0.1:46432 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
W20260929 01:57:33.991479 3113 connection.cc:573] server connection from 127.0.0.1:46126 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:57:33.992909 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:36413
--webserver_interface=127.0.71.126
--webserver_port=40277
--builtin_ntp_servers=127.0.71.84:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:36413,127.0.71.125:46843,127.0.71.124:46285
--master_addresses=127.0.71.126:36413,127.0.71.125:46843,127.0.71.124:46285,127.0.71.123:38679 with env {}
W20260929 01:57:34.441411 3696 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:57:34.441789 3696 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:57:34.441870 3696 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:57:34.451047 3696 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:57:34.451217 3696 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:57:34.451314 3696 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:57:34.451403 3696 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:57:34.462756 3696 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:36413,127.0.71.125:46843,127.0.71.124:46285,127.0.71.123:38679
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:36413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=40277
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:57:34.464882 3696 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:57:34.467271 3696 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:57:34.479151 3701 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:34.479198 3704 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:34.479223 3702 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:57:34.480330 3696 server_base.cc:1061] running on GCE node
I20260929 01:57:34.481438 3696 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:57:34.483333 3696 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:57:34.486959 3696 hybrid_clock.cc:648] HybridClock initialized: now 1790647054486796 us; error 86 us; skew 500 ppm
I20260929 01:57:34.488068 3696 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:57:34.492403 3696 webserver.cc:533] Webserver started at http://127.0.71.126:40277/ using document root <none> and password file <none>
I20260929 01:57:34.493530 3696 fs_manager.cc:362] Metadata directory not provided
I20260929 01:57:34.493683 3696 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:57:34.501983 3696 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.003s
I20260929 01:57:34.506367 3710 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:34.508744 3696 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:57:34.509107 3696 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "5dd1d0fad1154df698b109127fc98e76"
format_stamp: "Formatted at 2026-09-29 01:57:29 on dist-test-slave-vz23"
I20260929 01:57:34.510174 3696 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:57:34.528064 3696 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:57:34.529250 3696 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:57:34.529642 3696 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:57:34.554951 3696 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:36413
I20260929 01:57:34.554951 3761 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:36413 every 8 connection(s)
I20260929 01:57:34.557225 3696 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:57:34.564477 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 3696
I20260929 01:57:34.564850 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 3553
I20260929 01:57:34.568548 3762 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:57:34.574055 3762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76: Bootstrap starting.
I20260929 01:57:34.577452 3676 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:57:34.598124 3762 log.cc:826] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76: Log is configured to *not* fsync() on all Append() calls
I20260929 01:57:34.610844 3762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:57:34.611570 3762 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76: Bootstrap complete.
I20260929 01:57:34.620836 3762 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } }
I20260929 01:57:34.622092 3762 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5dd1d0fad1154df698b109127fc98e76, State: Initialized, Role: FOLLOWER
I20260929 01:57:34.624045 3762 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } }
I20260929 01:57:34.625469 3768 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } } }
I20260929 01:57:34.625933 3768 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:34.627122 3762 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:57:34.641479 3779 catalog_manager.cc:1308] Loaded cluster ID: 4e17683b73c2438f9fcbc54fde2e73dd
I20260929 01:57:34.641834 3779 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76: loading cluster ID for follower catalog manager: success
I20260929 01:57:34.647028 3779 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76: acquiring CA information for follower catalog manager: success
I20260929 01:57:34.652637 3779 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:57:34.698307 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 3553
I20260929 01:57:34.711014 3696 master_runner.cc:264] No leader master found from master 5dd1d0fad1154df698b109127fc98e76
I20260929 01:57:34.711272 3696 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:57:34.721302 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:39481
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:36413,127.0.71.125:46843,127.0.71.124:46285
--webserver_port=40881
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:36413,127.0.71.125:46843,127.0.71.124:46285,127.0.71.123:38679 with env {}
I20260929 01:57:35.158299 3787 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dd1d0fad1154df698b109127fc98e76)
I20260929 01:57:35.158593 3787 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } }
I20260929 01:57:35.160434 3787 leader_election.cc:290] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dd1d0fad1154df698b109127fc98e76 (127.0.71.126:36413), 126e333b2eab482c8c57e9e4938169bd (127.0.71.124:46285)
I20260929 01:57:35.161324 3137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9167d627ecab4454a801c180b4cb0bf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "126e333b2eab482c8c57e9e4938169bd" is_pre_election: true
I20260929 01:57:35.170516 3737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9167d627ecab4454a801c180b4cb0bf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5dd1d0fad1154df698b109127fc98e76" is_pre_election: true
I20260929 01:57:35.171151 3737 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9167d627ecab4454a801c180b4cb0bf9 in term 1.
I20260929 01:57:35.172008 3039 leader_election.cc:304] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dd1d0fad1154df698b109127fc98e76, 9167d627ecab4454a801c180b4cb0bf9; no voters: 126e333b2eab482c8c57e9e4938169bd
I20260929 01:57:35.172590 3787 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 01:57:35.172777 3787 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [term 1 FOLLOWER]: Starting leader election (detected failure of leader 5dd1d0fad1154df698b109127fc98e76)
I20260929 01:57:35.172914 3787 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:57:35.175431 3787 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } }
I20260929 01:57:35.176400 3787 leader_election.cc:290] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [CANDIDATE]: Term 2 election: Requested vote from peers 5dd1d0fad1154df698b109127fc98e76 (127.0.71.126:36413), 126e333b2eab482c8c57e9e4938169bd (127.0.71.124:46285)
I20260929 01:57:35.177139 3737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9167d627ecab4454a801c180b4cb0bf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5dd1d0fad1154df698b109127fc98e76"
I20260929 01:57:35.177184 3137 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9167d627ecab4454a801c180b4cb0bf9" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "126e333b2eab482c8c57e9e4938169bd"
I20260929 01:57:35.177392 3737 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [term 1 FOLLOWER]: Advancing to term 2
W20260929 01:57:35.177500 3786 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:57:35.177994 3786 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:57:35.178192 3786 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260929 01:57:35.183444 3737 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9167d627ecab4454a801c180b4cb0bf9 in term 2.
I20260929 01:57:35.184084 3039 leader_election.cc:304] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 5dd1d0fad1154df698b109127fc98e76, 9167d627ecab4454a801c180b4cb0bf9; no voters: 126e333b2eab482c8c57e9e4938169bd
I20260929 01:57:35.184489 3787 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:57:35.184995 3787 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [term 2 LEADER]: Becoming Leader. State: Replica: 9167d627ecab4454a801c180b4cb0bf9, State: Running, Role: LEADER
I20260929 01:57:35.185554 3787 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } }
I20260929 01:57:35.187907 3792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9167d627ecab4454a801c180b4cb0bf9. Latest consensus state: current_term: 2 leader_uuid: "9167d627ecab4454a801c180b4cb0bf9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } } }
I20260929 01:57:35.188205 3792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [sys.catalog]: This master's current role is: LEADER
W20260929 01:57:35.188921 3786 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
I20260929 01:57:35.189030 3794 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
W20260929 01:57:35.189169 3786 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:57:35.191169 3794 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:57:35.192039 3794 catalog_manager.cc:1308] Loaded cluster ID: 4e17683b73c2438f9fcbc54fde2e73dd
I20260929 01:57:35.192169 3794 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:57:35.193436 3794 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:57:35.194365 3794 catalog_manager.cc:6230] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9: Loaded TSK: 0
I20260929 01:57:35.195379 3794 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:57:35.201335 3786 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:39481
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=40881
--tserver_master_addrs=127.0.71.126:36413,127.0.71.125:46843,127.0.71.124:46285,127.0.71.123:38679
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:57:35.203135 3786 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:57:35.205564 3786 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:57:35.219223 3796 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:35.219223 3797 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:35.219552 3799 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:57:35.220311 3786 server_base.cc:1061] running on GCE node
I20260929 01:57:35.221101 3786 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:57:35.222288 3786 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:57:35.223548 3786 hybrid_clock.cc:648] HybridClock initialized: now 1790647055223476 us; error 68 us; skew 500 ppm
I20260929 01:57:35.223959 3786 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:57:35.227140 3786 webserver.cc:533] Webserver started at http://127.0.71.65:40881/ using document root <none> and password file <none>
I20260929 01:57:35.228307 3786 fs_manager.cc:362] Metadata directory not provided
I20260929 01:57:35.228600 3786 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:57:35.236826 3786 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:57:35.240720 3805 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:35.242695 3786 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:57:35.242944 3786 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "1dc520dc7c1b4d46a8409d4577afcc0c"
format_stamp: "Formatted at 2026-09-29 01:57:31 on dist-test-slave-vz23"
I20260929 01:57:35.243810 3786 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:57:35.270017 3786 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:57:35.272158 3786 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:57:35.272646 3786 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:57:35.273892 3786 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:57:35.276171 3786 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:57:35.276310 3786 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:35.276535 3786 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:57:35.276618 3786 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.001s sys 0.000s
I20260929 01:57:35.319007 3786 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:39481
I20260929 01:57:35.319063 3917 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:39481 every 8 connection(s)
I20260929 01:57:35.321063 3786 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:57:35.324635 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 3786
I20260929 01:57:35.354480 3920 heartbeater.cc:344] Connected to a master server at 127.0.71.126:36413
I20260929 01:57:35.354992 3920 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:35.356586 3920 heartbeater.cc:507] Master 127.0.71.126:36413 requested a full tablet report, sending...
I20260929 01:57:35.360311 3727 ts_manager.cc:194] Registered new tserver with Master: 1dc520dc7c1b4d46a8409d4577afcc0c (127.0.71.65:39481)
I20260929 01:57:35.365396 3918 heartbeater.cc:344] Connected to a master server at 127.0.71.123:38679
I20260929 01:57:35.365641 3918 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:35.366395 3918 heartbeater.cc:507] Master 127.0.71.123:38679 requested a full tablet report, sending...
I20260929 01:57:35.368844 3499 ts_manager.cc:194] Re-registered known tserver with Master: 1dc520dc7c1b4d46a8409d4577afcc0c (127.0.71.65:39481)
I20260929 01:57:35.369400 3919 heartbeater.cc:344] Connected to a master server at 127.0.71.124:46285
I20260929 01:57:35.369607 3919 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:35.370317 3919 heartbeater.cc:507] Master 127.0.71.124:46285 requested a full tablet report, sending...
I20260929 01:57:35.373157 3921 heartbeater.cc:344] Connected to a master server at 127.0.71.125:46843
I20260929 01:57:35.373410 3921 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:35.374169 3921 heartbeater.cc:507] Master 127.0.71.125:46843 requested a full tablet report, sending...
I20260929 01:57:35.374431 3127 ts_manager.cc:194] Re-registered known tserver with Master: 1dc520dc7c1b4d46a8409d4577afcc0c (127.0.71.65:39481)
I20260929 01:57:35.375579 3055 ts_manager.cc:194] Re-registered known tserver with Master: 1dc520dc7c1b4d46a8409d4577afcc0c (127.0.71.65:39481)
I20260929 01:57:35.377645 3055 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:35605
I20260929 01:57:35.402261 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 3786
I20260929 01:57:35.420487 3913 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:57:35.470510 3934 raft_consensus.cc:493] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [term 1 FOLLOWER]: Starting pre-election (detected failure of leader 5dd1d0fad1154df698b109127fc98e76)
I20260929 01:57:35.470824 3934 raft_consensus.cc:515] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } }
I20260929 01:57:35.472486 3934 leader_election.cc:290] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 5dd1d0fad1154df698b109127fc98e76 (127.0.71.126:36413), 9167d627ecab4454a801c180b4cb0bf9 (127.0.71.125:46843)
I20260929 01:57:35.473398 3064 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "126e333b2eab482c8c57e9e4938169bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "9167d627ecab4454a801c180b4cb0bf9" is_pre_election: true
I20260929 01:57:35.480903 3737 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "126e333b2eab482c8c57e9e4938169bd" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "5dd1d0fad1154df698b109127fc98e76" is_pre_election: true
I20260929 01:57:35.481426 3737 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 126e333b2eab482c8c57e9e4938169bd in current term 2: Already voted for candidate 9167d627ecab4454a801c180b4cb0bf9 in this term.
I20260929 01:57:35.482446 3111 leader_election.cc:304] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 126e333b2eab482c8c57e9e4938169bd; no voters: 5dd1d0fad1154df698b109127fc98e76, 9167d627ecab4454a801c180b4cb0bf9
I20260929 01:57:35.482961 3934 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:57:35.485585 3934 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:57:35.522212 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 3786
I20260929 01:57:35.543411 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:39481
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:36413,127.0.71.125:46843,127.0.71.124:46285
--webserver_port=40881
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:36413,127.0.71.125:46843,127.0.71.124:46285,127.0.71.123:38679
--fault_before_start=1.0 with env {}
I20260929 01:57:35.593031 3509 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565 [term 1 LEARNER]: Advancing to term 2
I20260929 01:57:35.597074 3509 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565 [term 2 LEARNER]: Refusing update from remote peer 9167d627ecab4454a801c180b4cb0bf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20260929 01:57:35.598366 3792 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:57:35.603174 3937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9167d627ecab4454a801c180b4cb0bf9. Latest consensus state: current_term: 2 leader_uuid: "9167d627ecab4454a801c180b4cb0bf9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } } }
I20260929 01:57:35.603880 3937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565 [sys.catalog]: This master's current role is: LEARNER
I20260929 01:57:35.605281 3792 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9: attempting to promote NON_VOTER 742125c6ce0045198bfe0fe0331cb565 to VOTER
W20260929 01:57:35.605769 3792 raft_consensus.cc:1068] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9: Unable to promote non-voter 742125c6ce0045198bfe0fe0331cb565: Illegal state: Leader has not yet committed an operation in its own term
I20260929 01:57:35.707743 3137 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [term 2 FOLLOWER]: Refusing update from remote peer 9167d627ecab4454a801c180b4cb0bf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20260929 01:57:35.708573 3792 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:57:35.711150 3934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9167d627ecab4454a801c180b4cb0bf9. Latest consensus state: current_term: 2 leader_uuid: "9167d627ecab4454a801c180b4cb0bf9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } } }
I20260929 01:57:35.711462 3934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:35.714404 3787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "9167d627ecab4454a801c180b4cb0bf9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } } }
I20260929 01:57:35.714478 3792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "9167d627ecab4454a801c180b4cb0bf9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } } }
I20260929 01:57:35.714670 3787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [sys.catalog]: This master's current role is: LEADER
I20260929 01:57:35.714675 3792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [sys.catalog]: This master's current role is: LEADER
I20260929 01:57:35.714926 3737 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [term 2 FOLLOWER]: Refusing update from remote peer 9167d627ecab4454a801c180b4cb0bf9: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20260929 01:57:35.715721 3792 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:57:35.715654 3934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "9167d627ecab4454a801c180b4cb0bf9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } } }
I20260929 01:57:35.715920 3934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:35.718627 3937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "9167d627ecab4454a801c180b4cb0bf9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } } }
I20260929 01:57:35.718892 3937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565 [sys.catalog]: This master's current role is: LEARNER
I20260929 01:57:35.720172 3941 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9: attempting to promote NON_VOTER 742125c6ce0045198bfe0fe0331cb565 to VOTER
I20260929 01:57:35.721525 3941 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: false } }
I20260929 01:57:35.725116 3137 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [term 2 FOLLOWER]: Refusing update from remote peer 9167d627ecab4454a801c180b4cb0bf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20260929 01:57:35.725185 3509 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565 [term 2 LEARNER]: Refusing update from remote peer 9167d627ecab4454a801c180b4cb0bf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20260929 01:57:35.726670 3941 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 01:57:35.727639 3792 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 01:57:35.732708 3736 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [term 2 FOLLOWER]: Refusing update from remote peer 9167d627ecab4454a801c180b4cb0bf9: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20260929 01:57:35.732941 3944 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9167d627ecab4454a801c180b4cb0bf9. Latest consensus state: current_term: 2 leader_uuid: "9167d627ecab4454a801c180b4cb0bf9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } } }
I20260929 01:57:35.733320 3944 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:35.732900 3792 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 742125c6ce0045198bfe0fe0331cb565 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: false } } }
I20260929 01:57:35.733922 3787 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [LEADER]: Connected to new peer: Peer: permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 01:57:35.733922 3948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "9167d627ecab4454a801c180b4cb0bf9" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: true } } }
I20260929 01:57:35.734292 3948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:35.735280 3137 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 742125c6ce0045198bfe0fe0331cb565 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: false } } }
I20260929 01:57:35.736034 3736 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 742125c6ce0045198bfe0fe0331cb565 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: false } } }
I20260929 01:57:35.739959 3934 sys_catalog.cc:455] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "9167d627ecab4454a801c180b4cb0bf9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: false } } }
I20260929 01:57:35.740242 3934 sys_catalog.cc:458] T 00000000000000000000000000000000 P 126e333b2eab482c8c57e9e4938169bd [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:35.741638 3948 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "9167d627ecab4454a801c180b4cb0bf9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: false } } }
I20260929 01:57:35.741899 3948 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5dd1d0fad1154df698b109127fc98e76 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:57:35.744496 3787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 2 leader_uuid: "9167d627ecab4454a801c180b4cb0bf9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: false } } }
I20260929 01:57:35.744704 3787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [sys.catalog]: This master's current role is: LEADER
I20260929 01:57:35.744419 3792 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "9167d627ecab4454a801c180b4cb0bf9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: false } } }
I20260929 01:57:35.744995 3792 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [sys.catalog]: This master's current role is: LEADER
I20260929 01:57:35.745319 3787 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "9167d627ecab4454a801c180b4cb0bf9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: false } } }
I20260929 01:57:35.745633 3787 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [sys.catalog]: This master's current role is: LEADER
I20260929 01:57:35.746075 3949 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "9167d627ecab4454a801c180b4cb0bf9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: false } } }
I20260929 01:57:35.745461 3509 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, 742125c6ce0045198bfe0fe0331cb565 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: false } } }
I20260929 01:57:35.746343 3949 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9167d627ecab4454a801c180b4cb0bf9 [sys.catalog]: This master's current role is: LEADER
I20260929 01:57:35.749310 3937 sys_catalog.cc:455] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "9167d627ecab4454a801c180b4cb0bf9" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "5dd1d0fad1154df698b109127fc98e76" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36413 } } peers { permanent_uuid: "9167d627ecab4454a801c180b4cb0bf9" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 46843 } } peers { permanent_uuid: "126e333b2eab482c8c57e9e4938169bd" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 46285 } } peers { permanent_uuid: "742125c6ce0045198bfe0fe0331cb565" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 38679 } attrs { promote: false } } }
I20260929 01:57:35.749591 3937 sys_catalog.cc:458] T 00000000000000000000000000000000 P 742125c6ce0045198bfe0fe0331cb565 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:57:35.981118 3936 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:57:35.981575 3936 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:57:35.981731 3936 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:57:35.981839 3936 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:57:35.991226 3936 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:57:35.991564 3936 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:57:36.004976 3936 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:39481
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=40881
--tserver_master_addrs=127.0.71.126:36413,127.0.71.125:46843,127.0.71.124:46285,127.0.71.123:38679
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:57:36.007028 3936 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:57:36.009436 3936 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:57:36.022529 3954 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:36.022681 3955 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:36.023052 3957 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:57:36.023475 3936 server_base.cc:1061] running on GCE node
I20260929 01:57:36.024318 3936 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:57:36.025627 3936 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:57:36.026986 3936 hybrid_clock.cc:648] HybridClock initialized: now 1790647056026900 us; error 52 us; skew 500 ppm
I20260929 01:57:36.027436 3936 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:57:36.030637 3936 webserver.cc:533] Webserver started at http://127.0.71.65:40881/ using document root <none> and password file <none>
I20260929 01:57:36.031672 3936 fs_manager.cc:362] Metadata directory not provided
I20260929 01:57:36.031884 3936 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:57:36.039486 3936 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:57:36.044091 3963 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:36.046499 3936 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260929 01:57:36.046870 3936 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "1dc520dc7c1b4d46a8409d4577afcc0c"
format_stamp: "Formatted at 2026-09-29 01:57:31 on dist-test-slave-vz23"
I20260929 01:57:36.048117 3936 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:57:36.081800 3936 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:57:36.083112 3936 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:57:36.083482 3936 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:57:36.084786 3936 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:57:36.087026 3936 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:57:36.087208 3936 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:36.087414 3936 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:57:36.087513 3936 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:57:36.088014 3936 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:57:36.088178 3936 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:57:36.088279 3936 fault_injection.cc:59] injecting fault for kudu (pid 3936): FLAGS_fault_before_start (process will exit)
I20260929 01:57:36.109637 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:39481
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:36413,127.0.71.125:46843,127.0.71.124:46285
--webserver_port=40881
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:39667
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:36413,127.0.71.125:46843,127.0.71.124:46285,127.0.71.123:38679 with env {}
W20260929 01:57:36.521454 3971 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:57:36.521889 3971 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:57:36.522065 3971 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:57:36.531852 3971 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:57:36.532116 3971 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:57:36.543872 3971 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:39667
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:39481
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=40881
--tserver_master_addrs=127.0.71.126:36413,127.0.71.125:46843,127.0.71.124:46285,127.0.71.123:38679
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:57:36.546111 3971 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:57:36.548530 3971 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:57:36.560518 3977 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:36.560559 3976 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:57:36.563268 3979 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:57:36.564320 3971 server_base.cc:1061] running on GCE node
I20260929 01:57:36.565430 3971 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:57:36.566845 3971 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:57:36.568132 3971 hybrid_clock.cc:648] HybridClock initialized: now 1790647056568079 us; error 51 us; skew 500 ppm
I20260929 01:57:36.568744 3971 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:57:36.571555 3971 webserver.cc:533] Webserver started at http://127.0.71.65:40881/ using document root <none> and password file <none>
I20260929 01:57:36.572701 3971 fs_manager.cc:362] Metadata directory not provided
I20260929 01:57:36.572887 3971 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:57:36.580199 3971 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:57:36.584391 3985 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:36.587019 3971 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260929 01:57:36.587275 3971 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "1dc520dc7c1b4d46a8409d4577afcc0c"
format_stamp: "Formatted at 2026-09-29 01:57:31 on dist-test-slave-vz23"
I20260929 01:57:36.588234 3971 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:57:36.614984 3971 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:57:36.616055 3971 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:57:36.616398 3971 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:57:36.617817 3971 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:57:36.620497 3971 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:57:36.620621 3971 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 01:57:36.620826 3971 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:57:36.620890 3971 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:57:36.666067 3971 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:39481
I20260929 01:57:36.666177 4097 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:39481 every 8 connection(s)
I20260929 01:57:36.668251 3971 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:57:36.675957 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 3971
I20260929 01:57:36.676369 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 3971
I20260929 01:57:36.700285 4101 heartbeater.cc:344] Connected to a master server at 127.0.71.125:46843
I20260929 01:57:36.700992 4101 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:36.702761 4101 heartbeater.cc:507] Master 127.0.71.125:46843 requested a full tablet report, sending...
I20260929 01:57:36.705844 3055 ts_manager.cc:194] Re-registered known tserver with Master: 1dc520dc7c1b4d46a8409d4577afcc0c (127.0.71.65:39481)
I20260929 01:57:36.707167 3055 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:58267
I20260929 01:57:36.708837 4099 heartbeater.cc:344] Connected to a master server at 127.0.71.124:46285
I20260929 01:57:36.709060 4099 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:36.709851 4099 heartbeater.cc:507] Master 127.0.71.124:46285 requested a full tablet report, sending...
I20260929 01:57:36.711403 4100 heartbeater.cc:344] Connected to a master server at 127.0.71.126:36413
I20260929 01:57:36.711619 4100 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:36.712224 4100 heartbeater.cc:507] Master 127.0.71.126:36413 requested a full tablet report, sending...
I20260929 01:57:36.713164 3127 ts_manager.cc:194] Re-registered known tserver with Master: 1dc520dc7c1b4d46a8409d4577afcc0c (127.0.71.65:39481)
I20260929 01:57:36.713912 3727 ts_manager.cc:194] Re-registered known tserver with Master: 1dc520dc7c1b4d46a8409d4577afcc0c (127.0.71.65:39481)
I20260929 01:57:36.716528 4093 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:57:36.816928 4098 heartbeater.cc:344] Connected to a master server at 127.0.71.123:38679
I20260929 01:57:36.817226 4098 heartbeater.cc:461] Registering TS with master...
I20260929 01:57:36.817757 4098 heartbeater.cc:507] Master 127.0.71.123:38679 requested a full tablet report, sending...
I20260929 01:57:36.818384 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 3971
I20260929 01:57:36.837462 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 3696
I20260929 01:57:36.851102 3757 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:57:36.971238 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 3696
I20260929 01:57:36.998618 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 3021
I20260929 01:57:37.000174 3085 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:57:37.140062 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 3021
W20260929 01:57:37.165650 3486 connection.cc:573] server connection from 127.0.0.1:46444 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:57:37.165903 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 3092
I20260929 01:57:37.167546 3157 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:57:37.295186 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 3092
I20260929 01:57:37.319680 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 3359
I20260929 01:57:37.321110 3529 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:57:37.464824 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 3359
Shutting down hive metastore.
2026-09-29T01:57:37Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestAddMaster/4 (30146 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/5
2026-09-29T01:57:37Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:57:37Z Disabled control of system clock
2026-09-29T01:57:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:57:38Z Disabled control of system clock
2026-09-29T01:57:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:57:38Z Disabled control of system clock
2026-09-29T01:57:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:57:38Z Disabled control of system clock
2026-09-29T01:57:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:57:38Z Disabled control of system clock
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/security.properties
2026-09-29 01:57:48: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/security.properties
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
I20260929 01:57:59.585295 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:43673
--webserver_interface=127.0.71.126
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:41741,127.0.71.85:40765,127.0.71.86:42285,127.0.71.87:45011,127.0.71.88:39997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43673,127.0.71.125:44145,127.0.71.124:34843 with env {}
W20260929 01:57:59.984565 4409 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:57:59.985036 4409 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:57:59.985180 4409 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:57:59.996346 4409 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:57:59.996603 4409 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:57:59.996730 4409 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:57:59.996838 4409 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:58:00.009748 4409 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:41741,127.0.71.85:40765,127.0.71.86:42285,127.0.71.87:45011,127.0.71.88:39997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43673,127.0.71.125:44145,127.0.71.124:34843
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:43673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:00.012626 4409 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:00.016603 4409 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:00.028987 4415 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:00.029099 4414 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:00.029037 4417 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:00.030026 4409 server_base.cc:1061] running on GCE node
I20260929 01:58:00.031126 4409 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:00.033440 4409 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:00.037053 4409 hybrid_clock.cc:648] HybridClock initialized: now 1790647080036755 us; error 156 us; skew 500 ppm
I20260929 01:58:00.037771 4409 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:58:00.041085 4409 webserver.cc:533] Webserver started at http://127.0.71.126:36517/ using document root <none> and password file <none>
I20260929 01:58:00.042079 4409 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:00.042207 4409 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:00.042599 4409 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:58:00.046036 4409 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "aec8b1c2b2dd429e9314313ae5b15509"
format_stamp: "Formatted at 2026-09-29 01:58:00 on dist-test-slave-vz23"
I20260929 01:58:00.047235 4409 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "aec8b1c2b2dd429e9314313ae5b15509"
format_stamp: "Formatted at 2026-09-29 01:58:00 on dist-test-slave-vz23"
I20260929 01:58:00.056219 4409 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.003s
I20260929 01:58:00.062196 4423 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:00.064272 4409 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.001s
I20260929 01:58:00.064797 4409 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "aec8b1c2b2dd429e9314313ae5b15509"
format_stamp: "Formatted at 2026-09-29 01:58:00 on dist-test-slave-vz23"
I20260929 01:58:00.065089 4409 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:00.087939 4409 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:00.089219 4409 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:00.089532 4409 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:00.117730 4409 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:43673
I20260929 01:58:00.117763 4474 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:43673 every 8 connection(s)
I20260929 01:58:00.119984 4409 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:58:00.121309 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 4409
I20260929 01:58:00.121542 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:58:00.122792 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:44145
--webserver_interface=127.0.71.125
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:41741,127.0.71.85:40765,127.0.71.86:42285,127.0.71.87:45011,127.0.71.88:39997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43673,127.0.71.125:44145,127.0.71.124:34843 with env {}
I20260929 01:58:00.126386 4475 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:58:00.134516 4475 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:58:00.167078 4475 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:58:00.169078 4427 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.125:44145: connect: Connection refused (error 111)
W20260929 01:58:00.171838 4475 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:44145: Network error: Client connection negotiation failed: client connection to 127.0.71.125:44145: connect: Connection refused (error 111)
I20260929 01:58:00.221657 4475 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } attempt: 1
W20260929 01:58:00.224365 4475 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:44145: Network error: Client connection negotiation failed: client connection to 127.0.71.125:44145: connect: Connection refused (error 111)
I20260929 01:58:00.292865 4475 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } attempt: 2
W20260929 01:58:00.296320 4475 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:44145: Network error: Client connection negotiation failed: client connection to 127.0.71.125:44145: connect: Connection refused (error 111)
I20260929 01:58:00.387868 4475 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } attempt: 3
W20260929 01:58:00.390897 4475 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:44145: Network error: Client connection negotiation failed: client connection to 127.0.71.125:44145: connect: Connection refused (error 111)
I20260929 01:58:00.534284 4475 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } attempt: 4
W20260929 01:58:00.537413 4475 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:44145: Network error: Client connection negotiation failed: client connection to 127.0.71.125:44145: connect: Connection refused (error 111)
W20260929 01:58:00.578703 4477 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:00.579166 4477 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:00.579341 4477 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:00.589313 4477 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:58:00.589512 4477 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:00.589636 4477 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:58:00.589730 4477 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:58:00.601652 4477 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:41741,127.0.71.85:40765,127.0.71.86:42285,127.0.71.87:45011,127.0.71.88:39997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43673,127.0.71.125:44145,127.0.71.124:34843
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:44145
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:00.603876 4477 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:00.606222 4477 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:00.617805 4486 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:00.617947 4485 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:00.619256 4488 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:00.619419 4477 server_base.cc:1061] running on GCE node
I20260929 01:58:00.621002 4477 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:00.623307 4477 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:00.627036 4477 hybrid_clock.cc:648] HybridClock initialized: now 1790647080626776 us; error 180 us; skew 500 ppm
I20260929 01:58:00.628123 4477 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:58:00.634156 4477 webserver.cc:533] Webserver started at http://127.0.71.125:33295/ using document root <none> and password file <none>
I20260929 01:58:00.635257 4477 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:00.635470 4477 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:00.635960 4477 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:58:00.639245 4477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-1/data/instance:
uuid: "ccb82b4ac1f54419ae571dffc292927d"
format_stamp: "Formatted at 2026-09-29 01:58:00 on dist-test-slave-vz23"
I20260929 01:58:00.640575 4477 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-1/wal/instance:
uuid: "ccb82b4ac1f54419ae571dffc292927d"
format_stamp: "Formatted at 2026-09-29 01:58:00 on dist-test-slave-vz23"
I20260929 01:58:00.648989 4477 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.001s
I20260929 01:58:00.654496 4494 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:00.658179 4477 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.001s
I20260929 01:58:00.658536 4477 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "ccb82b4ac1f54419ae571dffc292927d"
format_stamp: "Formatted at 2026-09-29 01:58:00 on dist-test-slave-vz23"
I20260929 01:58:00.658847 4477 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:00.687745 4477 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:00.689020 4477 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:00.689452 4477 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:00.717727 4545 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:44145 every 8 connection(s)
I20260929 01:58:00.717727 4477 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:44145
I20260929 01:58:00.719717 4477 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:58:00.723409 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 4477
I20260929 01:58:00.723640 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:58:00.725139 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:34843
--webserver_interface=127.0.71.124
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:41741,127.0.71.85:40765,127.0.71.86:42285,127.0.71.87:45011,127.0.71.88:39997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43673,127.0.71.125:44145,127.0.71.124:34843 with env {}
I20260929 01:58:00.726334 4546 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:58:00.737393 4546 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:58:00.756093 4546 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:58:00.768867 4546 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:58:00.770442 4495 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.124:34843: connect: Connection refused (error 111)
W20260929 01:58:00.772557 4546 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:34843: Network error: Client connection negotiation failed: client connection to 127.0.71.124:34843: connect: Connection refused (error 111)
I20260929 01:58:00.822075 4546 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } attempt: 1
W20260929 01:58:00.826781 4546 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:34843: Network error: Client connection negotiation failed: client connection to 127.0.71.124:34843: connect: Connection refused (error 111)
I20260929 01:58:00.836855 4475 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } attempt: 5
I20260929 01:58:00.848662 4475 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:58:00.852684 4475 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:34843: Network error: Client connection negotiation failed: client connection to 127.0.71.124:34843: connect: Connection refused (error 111)
I20260929 01:58:00.895347 4546 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } attempt: 2
W20260929 01:58:00.900153 4546 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:34843: Network error: Client connection negotiation failed: client connection to 127.0.71.124:34843: connect: Connection refused (error 111)
I20260929 01:58:00.904186 4475 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } attempt: 1
W20260929 01:58:00.908095 4475 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:34843: Network error: Client connection negotiation failed: client connection to 127.0.71.124:34843: connect: Connection refused (error 111)
I20260929 01:58:00.976423 4475 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } attempt: 2
W20260929 01:58:00.980019 4475 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:34843: Network error: Client connection negotiation failed: client connection to 127.0.71.124:34843: connect: Connection refused (error 111)
I20260929 01:58:00.991528 4546 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } attempt: 3
W20260929 01:58:00.994855 4546 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:34843: Network error: Client connection negotiation failed: client connection to 127.0.71.124:34843: connect: Connection refused (error 111)
I20260929 01:58:01.086517 4475 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } attempt: 3
W20260929 01:58:01.090783 4475 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:34843: Network error: Client connection negotiation failed: client connection to 127.0.71.124:34843: connect: Connection refused (error 111)
I20260929 01:58:01.138283 4546 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } attempt: 4
W20260929 01:58:01.142400 4546 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:34843: Network error: Client connection negotiation failed: client connection to 127.0.71.124:34843: connect: Connection refused (error 111)
W20260929 01:58:01.239138 4548 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:01.239651 4548 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:01.239755 4548 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:01.249938 4548 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:58:01.250065 4548 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:01.250126 4548 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:58:01.250168 4548 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:58:01.261968 4548 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:41741,127.0.71.85:40765,127.0.71.86:42285,127.0.71.87:45011,127.0.71.88:39997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43673,127.0.71.125:44145,127.0.71.124:34843
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:34843
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:01.263944 4548 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:01.266455 4548 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260929 01:58:01.268165 4475 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } attempt: 4
W20260929 01:58:01.270864 4475 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:34843: Network error: Client connection negotiation failed: client connection to 127.0.71.124:34843: connect: Connection refused (error 111)
W20260929 01:58:01.278551 4557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:01.279085 4558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:01.280509 4560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:01.281814 4548 server_base.cc:1061] running on GCE node
I20260929 01:58:01.282963 4548 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:01.285317 4548 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:01.288925 4548 hybrid_clock.cc:648] HybridClock initialized: now 1790647081288709 us; error 122 us; skew 500 ppm
I20260929 01:58:01.289763 4548 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:58:01.293016 4548 webserver.cc:533] Webserver started at http://127.0.71.124:36615/ using document root <none> and password file <none>
I20260929 01:58:01.294015 4548 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:01.294215 4548 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:01.294647 4548 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:58:01.297878 4548 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-2/data/instance:
uuid: "640fd29e187441c28892247f391d462e"
format_stamp: "Formatted at 2026-09-29 01:58:01 on dist-test-slave-vz23"
I20260929 01:58:01.298949 4548 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-2/wal/instance:
uuid: "640fd29e187441c28892247f391d462e"
format_stamp: "Formatted at 2026-09-29 01:58:01 on dist-test-slave-vz23"
I20260929 01:58:01.306587 4548 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.001s sys 0.007s
I20260929 01:58:01.311949 4566 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:01.314461 4548 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260929 01:58:01.314774 4548 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "640fd29e187441c28892247f391d462e"
format_stamp: "Formatted at 2026-09-29 01:58:01 on dist-test-slave-vz23"
I20260929 01:58:01.315071 4548 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:01.365206 4548 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:01.366786 4548 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:01.367803 4548 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:01.402674 4548 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:34843
I20260929 01:58:01.402664 4617 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:34843 every 8 connection(s)
I20260929 01:58:01.404374 4548 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:58:01.410966 4618 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:58:01.413363 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 4548
I20260929 01:58:01.413913 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:58:01.424346 4618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:58:01.441924 4546 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } attempt: 5
I20260929 01:58:01.449909 4618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:58:01.461480 4618 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:58:01.462787 4546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d: Bootstrap starting.
I20260929 01:58:01.468724 4546 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d: Neither blocks nor log segments found. Creating new log.
I20260929 01:58:01.470880 4546 log.cc:826] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:58:01.475286 4546 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d: No bootstrap required, opened a new log
I20260929 01:58:01.477034 4618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e: Bootstrap starting.
I20260929 01:58:01.481459 4618 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e: Neither blocks nor log segments found. Creating new log.
I20260929 01:58:01.483150 4546 raft_consensus.cc:359] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } }
I20260929 01:58:01.483613 4546 raft_consensus.cc:385] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:58:01.483511 4618 log.cc:826] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e: Log is configured to *not* fsync() on all Append() calls
I20260929 01:58:01.483801 4546 raft_consensus.cc:740] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ccb82b4ac1f54419ae571dffc292927d, State: Initialized, Role: FOLLOWER
I20260929 01:58:01.484802 4546 consensus_queue.cc:260] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } }
I20260929 01:58:01.486223 4628 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } }
I20260929 01:58:01.486601 4628 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:01.487983 4546 sys_catalog.cc:565] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:58:01.488159 4618 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e: No bootstrap required, opened a new log
I20260929 01:58:01.497987 4618 raft_consensus.cc:359] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } }
I20260929 01:58:01.498497 4618 raft_consensus.cc:385] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:58:01.498692 4618 raft_consensus.cc:740] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 640fd29e187441c28892247f391d462e, State: Initialized, Role: FOLLOWER
I20260929 01:58:01.499816 4618 consensus_queue.cc:260] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } }
W20260929 01:58:01.502637 4639 catalog_manager.cc:1607] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:58:01.502956 4639 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:58:01.506461 4641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } }
I20260929 01:58:01.507194 4641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:01.499176 4477 master_runner.cc:187] Error getting master registration for 127.0.71.126:43673: OK, instance_id { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" instance_seqno: 1790647080109395 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:58:01.508292 4618 sys_catalog.cc:565] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:58:01.508688 4477 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:58:01.530051 4652 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:58:01.530392 4652 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:58:01.521134 4548 master_runner.cc:187] Error getting master registration for 127.0.71.126:43673: OK, instance_id { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" instance_seqno: 1790647080109395 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:58:01.530884 4548 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:58:01.548290 4475 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } attempt: 5
I20260929 01:58:01.569748 4475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509: Bootstrap starting.
I20260929 01:58:01.577718 4475 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509: Neither blocks nor log segments found. Creating new log.
I20260929 01:58:01.580826 4475 log.cc:826] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509: Log is configured to *not* fsync() on all Append() calls
I20260929 01:58:01.589850 4475 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509: No bootstrap required, opened a new log
I20260929 01:58:01.600209 4475 raft_consensus.cc:359] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } }
I20260929 01:58:01.600893 4475 raft_consensus.cc:385] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:58:01.601120 4475 raft_consensus.cc:740] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec8b1c2b2dd429e9314313ae5b15509, State: Initialized, Role: FOLLOWER
I20260929 01:58:01.602149 4475 consensus_queue.cc:260] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } }
I20260929 01:58:01.607196 4655 sys_catalog.cc:455] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } }
I20260929 01:58:01.608347 4655 sys_catalog.cc:458] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:01.609668 4475 sys_catalog.cc:565] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:58:01.621737 4572 tablet.cc:2433] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:58:01.646123 4666 catalog_manager.cc:1607] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260929 01:58:01.653712 4409 master_runner.cc:264] No leader master found from master aec8b1c2b2dd429e9314313ae5b15509
W20260929 01:58:01.655298 4666 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:58:01.655380 4409 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:58:01.693908 4500 tablet.cc:2433] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:58:01.724793 4628 raft_consensus.cc:493] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:58:01.725258 4628 raft_consensus.cc:515] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } }
I20260929 01:58:01.727618 4641 raft_consensus.cc:493] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:58:01.727998 4641 raft_consensus.cc:515] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } }
I20260929 01:58:01.730199 4641 leader_election.cc:290] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aec8b1c2b2dd429e9314313ae5b15509 (127.0.71.126:43673), ccb82b4ac1f54419ae571dffc292927d (127.0.71.125:44145)
I20260929 01:58:01.730496 4450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ccb82b4ac1f54419ae571dffc292927d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec8b1c2b2dd429e9314313ae5b15509" is_pre_election: true
I20260929 01:58:01.731273 4450 raft_consensus.cc:2468] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ccb82b4ac1f54419ae571dffc292927d in term 0.
I20260929 01:58:01.732486 4521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "640fd29e187441c28892247f391d462e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb82b4ac1f54419ae571dffc292927d" is_pre_election: true
I20260929 01:58:01.733078 4521 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 640fd29e187441c28892247f391d462e in term 0.
I20260929 01:58:01.734555 4570 leader_election.cc:304] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 640fd29e187441c28892247f391d462e, ccb82b4ac1f54419ae571dffc292927d; no voters:
I20260929 01:58:01.736304 4450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "640fd29e187441c28892247f391d462e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec8b1c2b2dd429e9314313ae5b15509" is_pre_election: true
I20260929 01:58:01.736668 4450 raft_consensus.cc:2468] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 640fd29e187441c28892247f391d462e in term 0.
I20260929 01:58:01.738404 4641 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:58:01.738806 4641 raft_consensus.cc:493] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:58:01.738988 4641 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:58:01.741865 4497 leader_election.cc:304] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: aec8b1c2b2dd429e9314313ae5b15509, ccb82b4ac1f54419ae571dffc292927d; no voters:
I20260929 01:58:01.744308 4593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ccb82b4ac1f54419ae571dffc292927d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "640fd29e187441c28892247f391d462e" is_pre_election: true
I20260929 01:58:01.745381 4628 leader_election.cc:290] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers aec8b1c2b2dd429e9314313ae5b15509 (127.0.71.126:43673), 640fd29e187441c28892247f391d462e (127.0.71.124:34843)
I20260929 01:58:01.746537 4628 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:58:01.751412 4641 raft_consensus.cc:515] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } }
I20260929 01:58:01.752753 4628 raft_consensus.cc:493] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:58:01.753520 4593 raft_consensus.cc:2393] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate ccb82b4ac1f54419ae571dffc292927d in current term 1: Already voted for candidate 640fd29e187441c28892247f391d462e in this term.
I20260929 01:58:01.754065 4628 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:58:01.757766 4450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "640fd29e187441c28892247f391d462e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec8b1c2b2dd429e9314313ae5b15509"
I20260929 01:58:01.758105 4450 raft_consensus.cc:3060] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:58:01.765645 4450 raft_consensus.cc:2468] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 640fd29e187441c28892247f391d462e in term 1.
I20260929 01:58:01.766429 4569 leader_election.cc:304] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 640fd29e187441c28892247f391d462e, aec8b1c2b2dd429e9314313ae5b15509; no voters:
I20260929 01:58:01.765659 4628 raft_consensus.cc:515] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } }
I20260929 01:58:01.773388 4669 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:58:01.774696 4593 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ccb82b4ac1f54419ae571dffc292927d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "640fd29e187441c28892247f391d462e"
I20260929 01:58:01.776413 4628 leader_election.cc:290] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [CANDIDATE]: Term 1 election: Requested vote from peers aec8b1c2b2dd429e9314313ae5b15509 (127.0.71.126:43673), 640fd29e187441c28892247f391d462e (127.0.71.124:34843)
I20260929 01:58:01.777325 4641 leader_election.cc:290] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [CANDIDATE]: Term 1 election: Requested vote from peers aec8b1c2b2dd429e9314313ae5b15509 (127.0.71.126:43673), ccb82b4ac1f54419ae571dffc292927d (127.0.71.125:44145)
I20260929 01:58:01.777818 4521 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "640fd29e187441c28892247f391d462e" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ccb82b4ac1f54419ae571dffc292927d"
I20260929 01:58:01.778697 4521 raft_consensus.cc:2393] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate 640fd29e187441c28892247f391d462e in current term 1: Already voted for candidate ccb82b4ac1f54419ae571dffc292927d in this term.
I20260929 01:58:01.784171 4669 raft_consensus.cc:697] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [term 1 LEADER]: Becoming Leader. State: Replica: 640fd29e187441c28892247f391d462e, State: Running, Role: LEADER
I20260929 01:58:01.785236 4669 consensus_queue.cc:237] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } }
W20260929 01:58:01.785655 4497 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.71.126:43673, user_credentials={real_user=slave}} blocked reactor thread for 43908.1us
I20260929 01:58:01.786841 4450 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ccb82b4ac1f54419ae571dffc292927d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "aec8b1c2b2dd429e9314313ae5b15509"
I20260929 01:58:01.787351 4450 raft_consensus.cc:2393] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [term 1 FOLLOWER]: Leader election vote request: Denying vote to candidate ccb82b4ac1f54419ae571dffc292927d in current term 1: Already voted for candidate 640fd29e187441c28892247f391d462e in this term.
I20260929 01:58:01.806037 4641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 640fd29e187441c28892247f391d462e. Latest consensus state: current_term: 1 leader_uuid: "640fd29e187441c28892247f391d462e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } }
I20260929 01:58:01.806386 4641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:01.808243 4495 leader_election.cc:304] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [CANDIDATE]: Term 1 election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: ccb82b4ac1f54419ae571dffc292927d; no voters: 640fd29e187441c28892247f391d462e, aec8b1c2b2dd429e9314313ae5b15509
I20260929 01:58:01.809023 4668 raft_consensus.cc:2749] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [term 1 FOLLOWER]: Leader election lost for term 1. Reason: could not achieve majority
I20260929 01:58:01.818580 4672 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:58:01.821940 4672 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:58:01.842674 4521 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [term 1 FOLLOWER]: Refusing update from remote peer 640fd29e187441c28892247f391d462e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
W20260929 01:58:01.844182 4429 tablet.cc:2433] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:58:01.846565 4641 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:58:01.856670 4450 raft_consensus.cc:1275] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [term 1 FOLLOWER]: Refusing update from remote peer 640fd29e187441c28892247f391d462e: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:58:01.861680 4641 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 01:58:01.889569 4655 sys_catalog.cc:455] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 640fd29e187441c28892247f391d462e. Latest consensus state: current_term: 1 leader_uuid: "640fd29e187441c28892247f391d462e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } }
I20260929 01:58:01.890004 4655 sys_catalog.cc:458] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:01.902474 4673 mvcc.cc:204] Tried to move back new op lower bound from 7334490447207550976 to 7334490447000199168. Current Snapshot: MvccSnapshot[applied={T|T < 7334490447207550976}]
I20260929 01:58:01.904639 4668 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 640fd29e187441c28892247f391d462e. Latest consensus state: current_term: 1 leader_uuid: "640fd29e187441c28892247f391d462e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } }
I20260929 01:58:01.905211 4668 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:01.905297 4670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "640fd29e187441c28892247f391d462e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } }
I20260929 01:58:01.905733 4670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:01.908981 4670 sys_catalog.cc:455] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "640fd29e187441c28892247f391d462e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } }
I20260929 01:58:01.909307 4670 sys_catalog.cc:458] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:01.918197 4672 catalog_manager.cc:1396] Generated new cluster ID: acce1065f8e846ea9dda9c9be6c88686
I20260929 01:58:01.918398 4672 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:58:01.924809 4668 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "640fd29e187441c28892247f391d462e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } }
I20260929 01:58:01.925505 4668 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:01.945048 4674 mvcc.cc:204] Tried to move back new op lower bound from 7334490447207550976 to 7334490447000199168. Current Snapshot: MvccSnapshot[applied={T|T < 7334490447207550976 or (T in {7334490447207550976})}]
I20260929 01:58:01.952162 4655 sys_catalog.cc:455] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "640fd29e187441c28892247f391d462e" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } }
I20260929 01:58:01.952531 4655 sys_catalog.cc:458] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:02.065688 4672 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 01:58:02.068059 4672 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:58:02.127847 4672 catalog_manager.cc:6219] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e: Generated new TSK 0
I20260929 01:58:02.129546 4672 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 01:58:02.150665 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:43673,127.0.71.125:44145,127.0.71.124:34843
--builtin_ntp_servers=127.0.71.84:41741,127.0.71.85:40765,127.0.71.86:42285,127.0.71.87:45011,127.0.71.88:39997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 01:58:02.506032 4639 catalog_manager.cc:1308] Loaded cluster ID: acce1065f8e846ea9dda9c9be6c88686
I20260929 01:58:02.506227 4639 catalog_manager.cc:1601] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d: loading cluster ID for follower catalog manager: success
I20260929 01:58:02.510188 4639 catalog_manager.cc:1623] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d: acquiring CA information for follower catalog manager: success
I20260929 01:58:02.513228 4639 catalog_manager.cc:1651] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:58:02.657929 4666 catalog_manager.cc:1308] Loaded cluster ID: acce1065f8e846ea9dda9c9be6c88686
I20260929 01:58:02.658102 4666 catalog_manager.cc:1601] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509: loading cluster ID for follower catalog manager: success
I20260929 01:58:02.661827 4666 catalog_manager.cc:1623] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509: acquiring CA information for follower catalog manager: success
I20260929 01:58:02.664772 4666 catalog_manager.cc:1651] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 01:58:02.726791 4681 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:02.727257 4681 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:02.727490 4681 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:02.738752 4681 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:02.739048 4681 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:58:02.752102 4681 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:41741,127.0.71.85:40765,127.0.71.86:42285,127.0.71.87:45011,127.0.71.88:39997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:43673,127.0.71.125:44145,127.0.71.124:34843
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:02.754295 4681 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:02.757140 4681 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:02.772223 4687 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:02.772269 4689 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:02.772526 4686 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:02.773634 4681 server_base.cc:1061] running on GCE node
I20260929 01:58:02.774578 4681 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:02.776254 4681 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:02.779882 4681 hybrid_clock.cc:648] HybridClock initialized: now 1790647082779598 us; error 157 us; skew 500 ppm
I20260929 01:58:02.780521 4681 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:58:02.783711 4681 webserver.cc:533] Webserver started at http://127.0.71.65:35021/ using document root <none> and password file <none>
I20260929 01:58:02.784807 4681 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:02.784996 4681 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:02.785449 4681 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:58:02.789860 4681 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "7bdce4d380404d9ca379b1bb9a64a29d"
format_stamp: "Formatted at 2026-09-29 01:58:02 on dist-test-slave-vz23"
I20260929 01:58:02.791668 4681 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "7bdce4d380404d9ca379b1bb9a64a29d"
format_stamp: "Formatted at 2026-09-29 01:58:02 on dist-test-slave-vz23"
I20260929 01:58:02.802664 4681 fs_manager.cc:696] Time spent creating directory manager: real 0.010s user 0.008s sys 0.000s
I20260929 01:58:02.809876 4695 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:02.812657 4681 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.006s sys 0.001s
I20260929 01:58:02.812975 4681 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "7bdce4d380404d9ca379b1bb9a64a29d"
format_stamp: "Formatted at 2026-09-29 01:58:02 on dist-test-slave-vz23"
I20260929 01:58:02.813354 4681 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:02.848919 4681 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:02.850389 4681 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:02.850787 4681 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:02.853875 4681 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:58:02.856777 4681 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:58:02.856928 4681 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:02.857133 4681 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:58:02.857268 4681 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:02.919724 4681 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:42311
I20260929 01:58:02.919864 4807 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:42311 every 8 connection(s)
I20260929 01:58:02.921988 4681 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:58:02.929438 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 4681
I20260929 01:58:02.929715 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
I20260929 01:58:02.970916 4810 heartbeater.cc:344] Connected to a master server at 127.0.71.125:44145
I20260929 01:58:02.971860 4810 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:02.972077 4809 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43673
I20260929 01:58:02.972417 4809 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:02.973914 4808 heartbeater.cc:344] Connected to a master server at 127.0.71.124:34843
I20260929 01:58:02.974255 4808 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:02.974721 4809 heartbeater.cc:507] Master 127.0.71.126:43673 requested a full tablet report, sending...
I20260929 01:58:02.974983 4810 heartbeater.cc:507] Master 127.0.71.125:44145 requested a full tablet report, sending...
I20260929 01:58:02.976480 4808 heartbeater.cc:507] Master 127.0.71.124:34843 requested a full tablet report, sending...
I20260929 01:58:02.980281 4511 ts_manager.cc:194] Registered new tserver with Master: 7bdce4d380404d9ca379b1bb9a64a29d (127.0.71.65:42311)
I20260929 01:58:02.980342 4440 ts_manager.cc:194] Registered new tserver with Master: 7bdce4d380404d9ca379b1bb9a64a29d (127.0.71.65:42311)
I20260929 01:58:02.981748 4583 ts_manager.cc:194] Registered new tserver with Master: 7bdce4d380404d9ca379b1bb9a64a29d (127.0.71.65:42311)
I20260929 01:58:02.985064 4583 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:59517
I20260929 01:58:03.011018 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:58:03.014910 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.123:33471
--webserver_interface=127.0.71.123
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:41741,127.0.71.85:40765,127.0.71.86:42285,127.0.71.87:45011,127.0.71.88:39997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43673,127.0.71.125:44145,127.0.71.124:34843,127.0.71.123:33471 with env {}
W20260929 01:58:03.519124 4818 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:03.519733 4818 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:03.519899 4818 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:03.530453 4818 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:58:03.530699 4818 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:03.530805 4818 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:58:03.530944 4818 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:58:03.544871 4818 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:41741,127.0.71.85:40765,127.0.71.86:42285,127.0.71.87:45011,127.0.71.88:39997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43673,127.0.71.125:44145,127.0.71.124:34843,127.0.71.123:33471
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.123:33471
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.0.71.123
--webserver_port=0
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:03.547060 4818 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:03.549690 4818 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:03.561896 4824 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:03.561862 4823 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:03.564323 4826 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:03.565375 4818 server_base.cc:1061] running on GCE node
I20260929 01:58:03.566717 4818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:03.569235 4818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:03.572911 4818 hybrid_clock.cc:648] HybridClock initialized: now 1790647083572624 us; error 204 us; skew 500 ppm
I20260929 01:58:03.573568 4818 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:58:03.577840 4818 webserver.cc:533] Webserver started at http://127.0.71.123:39043/ using document root <none> and password file <none>
I20260929 01:58:03.579032 4818 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:03.579174 4818 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:03.579659 4818 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:58:03.583005 4818 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-3/data/instance:
uuid: "80fc3a48610748f6ad5d3c8a84dab906"
format_stamp: "Formatted at 2026-09-29 01:58:03 on dist-test-slave-vz23"
I20260929 01:58:03.584103 4818 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-3/wal/instance:
uuid: "80fc3a48610748f6ad5d3c8a84dab906"
format_stamp: "Formatted at 2026-09-29 01:58:03 on dist-test-slave-vz23"
I20260929 01:58:03.591945 4818 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260929 01:58:03.597949 4832 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:03.600307 4818 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:58:03.600749 4818 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "80fc3a48610748f6ad5d3c8a84dab906"
format_stamp: "Formatted at 2026-09-29 01:58:03 on dist-test-slave-vz23"
I20260929 01:58:03.601135 4818 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:03.626948 4818 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:03.628288 4818 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:03.628724 4818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:03.659883 4818 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.123:33471
I20260929 01:58:03.659900 4883 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.123:33471 every 8 connection(s)
I20260929 01:58:03.662010 4818 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
I20260929 01:58:03.665395 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 4818
I20260929 01:58:03.670994 4884 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:58:03.681717 4884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:58:03.696133 4884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:58:03.707077 4884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:58:03.719195 4884 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33471 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:58:03.735319 4884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906: Bootstrap starting.
I20260929 01:58:03.740697 4884 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906: Neither blocks nor log segments found. Creating new log.
I20260929 01:58:03.742486 4884 log.cc:826] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906: Log is configured to *not* fsync() on all Append() calls
I20260929 01:58:03.748337 4884 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906: No bootstrap required, opened a new log
I20260929 01:58:03.754834 4884 raft_consensus.cc:359] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33471 } }
I20260929 01:58:03.755288 4884 raft_consensus.cc:385] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:58:03.755379 4884 raft_consensus.cc:740] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 80fc3a48610748f6ad5d3c8a84dab906, State: Initialized, Role: FOLLOWER
I20260929 01:58:03.756554 4884 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33471 } }
I20260929 01:58:03.757886 4893 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33471 } } }
I20260929 01:58:03.758677 4893 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:03.759809 4884 sys_catalog.cc:565] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:58:03.776654 4904 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:58:03.776906 4904 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:58:03.790145 4818 master_runner.cc:429] Detected that this master 80fc3a48610748f6ad5d3c8a84dab906 is joining an existing cluster
I20260929 01:58:03.790323 4818 catalog_manager.cc:1037] Shutting down the Catalog Manager
I20260929 01:58:03.791846 4818 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260929 01:58:03.792412 4818 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260929 01:58:03.792718 4818 tablet_replica.cc:333] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906: stopping tablet replica
I20260929 01:58:03.795424 4818 master_runner.cc:434] Initiating AddMaster RPC to add 127.0.71.123:33471
I20260929 01:58:03.849920 4583 catalog_manager.cc:7351] Initiating ChangeConfig request to add master 127.0.71.123:33471: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906"
member_type: NON_VOTER
last_known_addr {
host: "127.0.71.123"
port: 33471
}
attrs {
promote: true
}
}
dest_uuid: "640fd29e187441c28892247f391d462e"
cas_config_opid_index: -1
I20260929 01:58:03.852254 4583 consensus_queue.cc:237] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: true } }
I20260929 01:58:03.857429 4450 raft_consensus.cc:1275] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [term 1 FOLLOWER]: Refusing update from remote peer 640fd29e187441c28892247f391d462e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 01:58:03.858541 4521 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [term 1 FOLLOWER]: Refusing update from remote peer 640fd29e187441c28892247f391d462e: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 01:58:03.858400 4916 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260929 01:58:03.859221 4569 consensus_peers.cc:597] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e -> Peer 80fc3a48610748f6ad5d3c8a84dab906 (127.0.71.123:33471): Couldn't send request to peer 80fc3a48610748f6ad5d3c8a84dab906. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260929 01:58:03.859459 4916 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:58:03.863411 4916 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 80fc3a48610748f6ad5d3c8a84dab906 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: true } } }
I20260929 01:58:03.865819 4521 raft_consensus.cc:2955] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 80fc3a48610748f6ad5d3c8a84dab906 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: true } } }
I20260929 01:58:03.865917 4450 raft_consensus.cc:2955] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 80fc3a48610748f6ad5d3c8a84dab906 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: true } } }
I20260929 01:58:03.868817 4916 catalog_manager.cc:7355] Successfully completed master ChangeConfig request to add master 127.0.71.123:33471
I20260929 01:58:03.871027 4818 master.cc:562] Master@127.0.71.123:33471 shutting down...
I20260929 01:58:03.885013 4920 sys_catalog.cc:455] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "640fd29e187441c28892247f391d462e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: true } } }
I20260929 01:58:03.884430 4919 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "640fd29e187441c28892247f391d462e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: true } } }
I20260929 01:58:03.885488 4920 sys_catalog.cc:458] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:03.885592 4919 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:03.886304 4641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "640fd29e187441c28892247f391d462e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: true } } }
I20260929 01:58:03.886590 4641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:03.887393 4641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "640fd29e187441c28892247f391d462e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: true } } }
I20260929 01:58:03.887647 4641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:03.887912 4641 sys_catalog.cc:455] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "640fd29e187441c28892247f391d462e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: true } } }
I20260929 01:58:03.888140 4641 sys_catalog.cc:458] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: This master's current role is: LEADER
W20260929 01:58:03.916769 4836 connection.cc:466] server connection from 127.0.0.1:50684 torn down before Call kudu.master.MasterService.ListTables from 127.0.0.1:50684 (request call id 13) could send its response
I20260929 01:58:03.919660 4818 master.cc:584] Master@127.0.71.123:33471 shutdown complete.
I20260929 01:58:03.919904 4818 master_runner.cc:306] Clearing existing system tablet
I20260929 01:58:03.920387 4818 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:03.920576 4818 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:03.923816 4818 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260929 01:58:03.925907 4926 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:03.926509 4818 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:03.926857 4818 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:58:03.927038 4818 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "80fc3a48610748f6ad5d3c8a84dab906"
format_stamp: "Formatted at 2026-09-29 01:58:03 on dist-test-slave-vz23"
I20260929 01:58:03.931258 4818 ts_tablet_manager.cc:1962] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:58:03.938066 4818 ts_tablet_manager.cc:1975] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 01:58:03.938642 4818 log.cc:1199] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260929 01:58:03.939316 4818 ts_tablet_manager.cc:1996] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906: Deleting consensus metadata
I20260929 01:58:03.939662 4818 master_runner.cc:316] Copying system tablet from 127.0.71.124:34843
I20260929 01:58:03.944103 4818 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906: tablet copy: Beginning tablet copy session from remote peer at address 127.0.71.124:34843
I20260929 01:58:03.955085 4603 tablet_copy_service.cc:140] P 640fd29e187441c28892247f391d462e: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 80fc3a48610748f6ad5d3c8a84dab906 ({username='slave'} at 127.0.0.1:60540)
I20260929 01:58:03.955343 4603 tablet_copy_service.cc:161] P 640fd29e187441c28892247f391d462e: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 80fc3a48610748f6ad5d3c8a84dab906 at {username='slave'} at 127.0.0.1:60540: session id = 80fc3a48610748f6ad5d3c8a84dab906-00000000000000000000000000000000
I20260929 01:58:03.959390 4603 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 01:58:03.962962 4818 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 01:58:03.970655 4818 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906: tablet copy: Starting download of 0 data blocks...
I20260929 01:58:03.971215 4818 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906: tablet copy: Starting download of 1 WAL segments...
I20260929 01:58:03.975538 4818 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 01:58:03.981390 4603 tablet_copy_service.cc:342] P 640fd29e187441c28892247f391d462e: Request end of tablet copy session 80fc3a48610748f6ad5d3c8a84dab906-00000000000000000000000000000000 received from {username='slave'} at 127.0.0.1:60540
I20260929 01:58:03.981721 4603 tablet_copy_service.cc:434] P 640fd29e187441c28892247f391d462e: ending tablet copy session 80fc3a48610748f6ad5d3c8a84dab906-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 80fc3a48610748f6ad5d3c8a84dab906
I20260929 01:58:03.989681 4808 heartbeater.cc:499] Master 127.0.71.124:34843 was elected leader, sending a full tablet report...
W20260929 01:58:03.995810 4818 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260929 01:58:04.002928 4818 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:04.007053 4934 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:04.007203 4933 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:04.008973 4936 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:04.009578 4818 server_base.cc:1061] running on GCE node
I20260929 01:58:04.010051 4818 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:04.011312 4818 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:04.012720 4818 hybrid_clock.cc:648] HybridClock initialized: now 1790647084012538 us; error 109 us; skew 500 ppm
I20260929 01:58:04.013095 4818 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:58:04.015254 4818 webserver.cc:533] Webserver started at http://127.0.71.123:45587/ using document root <none> and password file <none>
I20260929 01:58:04.015700 4818 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:04.015861 4818 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:04.019068 4818 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.000s sys 0.002s
I20260929 01:58:04.022688 4942 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:04.023595 4818 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 01:58:04.023828 4818 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "80fc3a48610748f6ad5d3c8a84dab906"
format_stamp: "Formatted at 2026-09-29 01:58:03 on dist-test-slave-vz23"
I20260929 01:58:04.024107 4818 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:04.046237 4818 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:04.047214 4818 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:04.077776 4818 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.123:33471
I20260929 01:58:04.077828 4993 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.123:33471 every 8 connection(s)
I20260929 01:58:04.079154 4818 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
I20260929 01:58:04.083660 4994 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:58:04.086788 4994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906: Bootstrap starting.
I20260929 01:58:04.108970 4994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:58:04.109781 4994 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906: Bootstrap complete.
I20260929 01:58:04.111023 4994 raft_consensus.cc:359] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: true } }
I20260929 01:58:04.111449 4994 raft_consensus.cc:740] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 80fc3a48610748f6ad5d3c8a84dab906, State: Initialized, Role: LEARNER
I20260929 01:58:04.111783 4994 consensus_queue.cc:260] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: true } }
I20260929 01:58:04.112841 4997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: true } } }
I20260929 01:58:04.113257 4997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906 [sys.catalog]: This master's current role is: LEARNER
I20260929 01:58:04.113354 4994 sys_catalog.cc:565] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:58:04.127353 5008 catalog_manager.cc:1308] Loaded cluster ID: acce1065f8e846ea9dda9c9be6c88686
I20260929 01:58:04.127533 5008 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906: loading cluster ID for follower catalog manager: success
I20260929 01:58:04.134011 5008 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906: acquiring CA information for follower catalog manager: success
I20260929 01:58:04.136907 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 4681
I20260929 01:58:04.137730 5008 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:58:04.149681 4803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:58:04.268354 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 4681
I20260929 01:58:04.289469 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:42311
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:43673,127.0.71.125:44145,127.0.71.124:34843
--webserver_port=35021
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:41741,127.0.71.85:40765,127.0.71.86:42285,127.0.71.87:45011,127.0.71.88:39997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:43673,127.0.71.125:44145,127.0.71.124:34843,127.0.71.123:33471 with env {}
I20260929 01:58:04.357511 4969 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260929 01:58:04.358704 4997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 640fd29e187441c28892247f391d462e. Latest consensus state: current_term: 1 leader_uuid: "640fd29e187441c28892247f391d462e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: true } } }
I20260929 01:58:04.359086 4997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906 [sys.catalog]: This master's current role is: LEARNER
W20260929 01:58:04.750828 5012 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:04.751262 5012 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:04.751420 5012 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:04.761240 5012 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:04.761543 5012 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:58:04.774705 5012 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:41741,127.0.71.85:40765,127.0.71.86:42285,127.0.71.87:45011,127.0.71.88:39997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:42311
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=35021
--tserver_master_addrs=127.0.71.126:43673,127.0.71.125:44145,127.0.71.124:34843,127.0.71.123:33471
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:04.776929 5012 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:04.779235 5012 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:04.792092 5017 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:04.792092 5018 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:04.792732 5020 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:04.793988 5012 server_base.cc:1061] running on GCE node
I20260929 01:58:04.795028 5012 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:04.796657 5012 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:04.798249 5012 hybrid_clock.cc:648] HybridClock initialized: now 1790647084798062 us; error 130 us; skew 500 ppm
I20260929 01:58:04.798740 5012 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:58:04.801975 5012 webserver.cc:533] Webserver started at http://127.0.71.65:35021/ using document root <none> and password file <none>
I20260929 01:58:04.803473 5012 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:04.803736 5012 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:04.811532 5012 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:58:04.815634 5026 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:04.817656 5012 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:58:04.817896 5012 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "7bdce4d380404d9ca379b1bb9a64a29d"
format_stamp: "Formatted at 2026-09-29 01:58:02 on dist-test-slave-vz23"
I20260929 01:58:04.818816 5012 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:04.820997 4921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "640fd29e187441c28892247f391d462e" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: true } } }
I20260929 01:58:04.821278 4921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:04.823868 5027 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e: attempting to promote NON_VOTER 80fc3a48610748f6ad5d3c8a84dab906 to VOTER
I20260929 01:58:04.825171 5027 consensus_queue.cc:237] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: false } }
I20260929 01:58:04.828179 4521 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [term 1 FOLLOWER]: Refusing update from remote peer 640fd29e187441c28892247f391d462e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:58:04.828380 4969 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906 [term 1 LEARNER]: Refusing update from remote peer 640fd29e187441c28892247f391d462e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:58:04.829218 5027 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [LEADER]: Connected to new peer: Peer: permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:58:04.830381 4450 raft_consensus.cc:1275] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [term 1 FOLLOWER]: Refusing update from remote peer 640fd29e187441c28892247f391d462e: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:58:04.830046 4921 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [LEADER]: Connected to new peer: Peer: permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:58:04.832190 5027 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [LEADER]: Connected to new peer: Peer: permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:58:04.836004 5028 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 80fc3a48610748f6ad5d3c8a84dab906 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: false } } }
I20260929 01:58:04.839080 4969 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 80fc3a48610748f6ad5d3c8a84dab906 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: false } } }
I20260929 01:58:04.840495 4450 raft_consensus.cc:2955] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 80fc3a48610748f6ad5d3c8a84dab906 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: false } } }
I20260929 01:58:04.843509 4921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "640fd29e187441c28892247f391d462e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: false } } }
I20260929 01:58:04.843871 4921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:04.844666 4921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "640fd29e187441c28892247f391d462e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: false } } }
I20260929 01:58:04.844960 4921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:04.845683 4921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "640fd29e187441c28892247f391d462e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: false } } }
I20260929 01:58:04.845119 4521 raft_consensus.cc:2955] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 80fc3a48610748f6ad5d3c8a84dab906 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: false } } }
I20260929 01:58:04.846155 4921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:04.847059 4921 sys_catalog.cc:455] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "640fd29e187441c28892247f391d462e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: false } } }
I20260929 01:58:04.847455 4921 sys_catalog.cc:458] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:04.850693 4997 sys_catalog.cc:455] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "640fd29e187441c28892247f391d462e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: false } } }
I20260929 01:58:04.851073 4997 sys_catalog.cc:458] T 00000000000000000000000000000000 P 80fc3a48610748f6ad5d3c8a84dab906 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:04.857038 5035 sys_catalog.cc:455] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "640fd29e187441c28892247f391d462e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: false } } }
I20260929 01:58:04.857400 5035 sys_catalog.cc:458] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:04.857542 5012 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:04.857184 5036 sys_catalog.cc:455] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "640fd29e187441c28892247f391d462e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: false } } }
I20260929 01:58:04.858770 5012 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:04.859020 5036 sys_catalog.cc:458] T 00000000000000000000000000000000 P ccb82b4ac1f54419ae571dffc292927d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:04.859328 5012 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:04.861057 5012 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:58:04.863524 5012 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:58:04.863653 5012 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:04.863777 5012 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:58:04.863832 5012 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:04.913400 5012 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:42311
I20260929 01:58:04.913545 5144 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:42311 every 8 connection(s)
I20260929 01:58:04.915699 5012 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:58:04.924177 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 5012
I20260929 01:58:04.946374 5148 heartbeater.cc:344] Connected to a master server at 127.0.71.125:44145
I20260929 01:58:04.946803 5145 heartbeater.cc:344] Connected to a master server at 127.0.71.124:34843
I20260929 01:58:04.947221 5145 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:04.947990 5148 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:04.948130 5146 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43673
I20260929 01:58:04.948299 5146 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:04.949081 5147 heartbeater.cc:344] Connected to a master server at 127.0.71.123:33471
I20260929 01:58:04.949092 5148 heartbeater.cc:507] Master 127.0.71.125:44145 requested a full tablet report, sending...
I20260929 01:58:04.949160 5146 heartbeater.cc:507] Master 127.0.71.126:43673 requested a full tablet report, sending...
I20260929 01:58:04.949126 5145 heartbeater.cc:507] Master 127.0.71.124:34843 requested a full tablet report, sending...
I20260929 01:58:04.949314 5147 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:04.951179 5147 heartbeater.cc:507] Master 127.0.71.123:33471 requested a full tablet report, sending...
I20260929 01:58:04.952272 4583 ts_manager.cc:194] Re-registered known tserver with Master: 7bdce4d380404d9ca379b1bb9a64a29d (127.0.71.65:42311)
I20260929 01:58:04.952297 4440 ts_manager.cc:194] Re-registered known tserver with Master: 7bdce4d380404d9ca379b1bb9a64a29d (127.0.71.65:42311)
I20260929 01:58:04.952553 4511 ts_manager.cc:194] Re-registered known tserver with Master: 7bdce4d380404d9ca379b1bb9a64a29d (127.0.71.65:42311)
I20260929 01:58:04.953557 4959 ts_manager.cc:194] Registered new tserver with Master: 7bdce4d380404d9ca379b1bb9a64a29d (127.0.71.65:42311)
I20260929 01:58:04.953599 4583 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:52999
I20260929 01:58:04.965690 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:58:04.984817 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:58:04.985070 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 4409
I20260929 01:58:04.986696 4470 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:58:05.123672 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 4409
I20260929 01:58:05.152149 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:43673
--webserver_interface=127.0.71.126
--webserver_port=36517
--builtin_ntp_servers=127.0.71.84:41741,127.0.71.85:40765,127.0.71.86:42285,127.0.71.87:45011,127.0.71.88:39997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:43673,127.0.71.125:44145,127.0.71.124:34843
--master_addresses=127.0.71.126:43673,127.0.71.125:44145,127.0.71.124:34843,127.0.71.123:33471 with env {}
W20260929 01:58:05.430644 4569 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.126:43673: connect: Connection refused (error 111)
W20260929 01:58:05.432912 4569 consensus_peers.cc:597] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e -> Peer aec8b1c2b2dd429e9314313ae5b15509 (127.0.71.126:43673): Couldn't send request to peer aec8b1c2b2dd429e9314313ae5b15509. Status: Network error: Client connection negotiation failed: client connection to 127.0.71.126:43673: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260929 01:58:05.604936 5159 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:05.605347 5159 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:05.605439 5159 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:05.617504 5159 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:58:05.617740 5159 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:05.617849 5159 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:58:05.617944 5159 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:58:05.631507 5159 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:41741,127.0.71.85:40765,127.0.71.86:42285,127.0.71.87:45011,127.0.71.88:39997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:43673,127.0.71.125:44145,127.0.71.124:34843,127.0.71.123:33471
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:43673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=36517
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:05.633775 5159 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:05.636047 5159 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:05.647070 5166 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:05.647073 5167 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:05.647764 5169 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:05.648866 5159 server_base.cc:1061] running on GCE node
I20260929 01:58:05.649938 5159 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:05.652762 5159 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:05.656531 5159 hybrid_clock.cc:648] HybridClock initialized: now 1790647085656136 us; error 196 us; skew 500 ppm
I20260929 01:58:05.657316 5159 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:58:05.661603 5159 webserver.cc:533] Webserver started at http://127.0.71.126:36517/ using document root <none> and password file <none>
I20260929 01:58:05.662813 5159 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:05.663137 5159 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:05.672286 5159 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.005s sys 0.000s
I20260929 01:58:05.677409 5175 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:05.679777 5159 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.003s
I20260929 01:58:05.680143 5159 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "aec8b1c2b2dd429e9314313ae5b15509"
format_stamp: "Formatted at 2026-09-29 01:58:00 on dist-test-slave-vz23"
I20260929 01:58:05.681231 5159 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:05.725795 5159 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:05.727542 5159 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:05.728406 5159 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:05.759660 5159 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:43673
I20260929 01:58:05.759680 5226 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:43673 every 8 connection(s)
I20260929 01:58:05.762419 5159 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:58:05.771544 5227 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:58:05.772766 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 5159
I20260929 01:58:05.773118 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 5012
I20260929 01:58:05.776633 5227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509: Bootstrap starting.
I20260929 01:58:05.785779 5140 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:58:05.800200 5227 log.cc:826] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509: Log is configured to *not* fsync() on all Append() calls
I20260929 01:58:05.814317 5227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:58:05.815263 5227 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509: Bootstrap complete.
I20260929 01:58:05.824331 5227 raft_consensus.cc:359] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: false } }
I20260929 01:58:05.825464 5227 raft_consensus.cc:740] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: aec8b1c2b2dd429e9314313ae5b15509, State: Initialized, Role: FOLLOWER
I20260929 01:58:05.826493 5227 consensus_queue.cc:260] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: false } }
I20260929 01:58:05.827796 5233 sys_catalog.cc:455] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: false } } }
I20260929 01:58:05.828204 5233 sys_catalog.cc:458] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:05.829926 5227 sys_catalog.cc:565] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:58:05.847071 5244 catalog_manager.cc:1308] Loaded cluster ID: acce1065f8e846ea9dda9c9be6c88686
I20260929 01:58:05.847460 5244 catalog_manager.cc:1601] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509: loading cluster ID for follower catalog manager: success
I20260929 01:58:05.855965 5244 catalog_manager.cc:1623] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509: acquiring CA information for follower catalog manager: success
I20260929 01:58:05.862159 5244 catalog_manager.cc:1651] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:58:05.914669 5028 sys_catalog.cc:455] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "640fd29e187441c28892247f391d462e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: false } } }
I20260929 01:58:05.915475 5028 sys_catalog.cc:458] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:05.929234 5233 sys_catalog.cc:455] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 640fd29e187441c28892247f391d462e. Latest consensus state: current_term: 1 leader_uuid: "640fd29e187441c28892247f391d462e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: false } } }
I20260929 01:58:05.929590 5233 sys_catalog.cc:458] T 00000000000000000000000000000000 P aec8b1c2b2dd429e9314313ae5b15509 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:05.938606 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 5012
I20260929 01:58:05.958434 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:42311
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:43673,127.0.71.125:44145,127.0.71.124:34843
--webserver_port=35021
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:41741,127.0.71.85:40765,127.0.71.86:42285,127.0.71.87:45011,127.0.71.88:39997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:43673,127.0.71.125:44145,127.0.71.124:34843,127.0.71.123:33471 with env {}
I20260929 01:58:06.365556 5251 sys_catalog.cc:455] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "640fd29e187441c28892247f391d462e" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "aec8b1c2b2dd429e9314313ae5b15509" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 43673 } } peers { permanent_uuid: "ccb82b4ac1f54419ae571dffc292927d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 44145 } } peers { permanent_uuid: "640fd29e187441c28892247f391d462e" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 34843 } } peers { permanent_uuid: "80fc3a48610748f6ad5d3c8a84dab906" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33471 } attrs { promote: false } } }
I20260929 01:58:06.365963 5251 sys_catalog.cc:458] T 00000000000000000000000000000000 P 640fd29e187441c28892247f391d462e [sys.catalog]: This master's current role is: LEADER
W20260929 01:58:06.445014 5254 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:06.445452 5254 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:06.445559 5254 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:06.455200 5254 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:06.455437 5254 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:58:06.469166 5254 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:41741,127.0.71.85:40765,127.0.71.86:42285,127.0.71.87:45011,127.0.71.88:39997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:42311
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=35021
--tserver_master_addrs=127.0.71.126:43673,127.0.71.125:44145,127.0.71.124:34843,127.0.71.123:33471
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:06.471335 5254 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:06.473680 5254 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:06.487433 5260 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:06.487287 5259 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:06.487355 5262 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:06.489763 5254 server_base.cc:1061] running on GCE node
I20260929 01:58:06.490674 5254 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:06.492529 5254 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:06.496021 5254 hybrid_clock.cc:648] HybridClock initialized: now 1790647086495738 us; error 208 us; skew 500 ppm
I20260929 01:58:06.496650 5254 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:58:06.499857 5254 webserver.cc:533] Webserver started at http://127.0.71.65:35021/ using document root <none> and password file <none>
I20260929 01:58:06.501031 5254 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:06.501228 5254 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:06.509387 5254 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 01:58:06.513823 5270 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:06.516621 5254 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:58:06.516901 5254 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "7bdce4d380404d9ca379b1bb9a64a29d"
format_stamp: "Formatted at 2026-09-29 01:58:02 on dist-test-slave-vz23"
I20260929 01:58:06.518086 5254 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:06.554172 5254 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:06.555261 5254 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:06.555601 5254 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:06.557224 5254 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:58:06.559420 5254 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:58:06.559525 5254 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:06.559655 5254 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:58:06.559697 5254 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:06.605294 5254 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:42311
I20260929 01:58:06.605345 5382 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:42311 every 8 connection(s)
I20260929 01:58:06.607386 5254 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:58:06.614039 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 5254
I20260929 01:58:06.650524 5383 heartbeater.cc:344] Connected to a master server at 127.0.71.124:34843
I20260929 01:58:06.651134 5383 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:06.651589 5386 heartbeater.cc:344] Connected to a master server at 127.0.71.125:44145
I20260929 01:58:06.651785 5386 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:06.653255 5384 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43673
I20260929 01:58:06.653635 5384 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:06.654855 5384 heartbeater.cc:507] Master 127.0.71.126:43673 requested a full tablet report, sending...
I20260929 01:58:06.655218 5383 heartbeater.cc:507] Master 127.0.71.124:34843 requested a full tablet report, sending...
I20260929 01:58:06.655601 5386 heartbeater.cc:507] Master 127.0.71.125:44145 requested a full tablet report, sending...
I20260929 01:58:06.657377 4583 ts_manager.cc:194] Re-registered known tserver with Master: 7bdce4d380404d9ca379b1bb9a64a29d (127.0.71.65:42311)
I20260929 01:58:06.657842 4511 ts_manager.cc:194] Re-registered known tserver with Master: 7bdce4d380404d9ca379b1bb9a64a29d (127.0.71.65:42311)
I20260929 01:58:06.658826 4583 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:60239
I20260929 01:58:06.658921 5385 heartbeater.cc:344] Connected to a master server at 127.0.71.123:33471
I20260929 01:58:06.659137 5385 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:06.659786 5385 heartbeater.cc:507] Master 127.0.71.123:33471 requested a full tablet report, sending...
I20260929 01:58:06.661576 4959 ts_manager.cc:194] Re-registered known tserver with Master: 7bdce4d380404d9ca379b1bb9a64a29d (127.0.71.65:42311)
I20260929 01:58:06.661742 5192 ts_manager.cc:194] Registered new tserver with Master: 7bdce4d380404d9ca379b1bb9a64a29d (127.0.71.65:42311)
I20260929 01:58:06.694304 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 5254
I20260929 01:58:06.710991 5378 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:58:06.819020 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 5254
I20260929 01:58:06.839577 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:42311
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:43673,127.0.71.125:44145,127.0.71.124:34843
--webserver_port=35021
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:41741,127.0.71.85:40765,127.0.71.86:42285,127.0.71.87:45011,127.0.71.88:39997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:43673,127.0.71.125:44145,127.0.71.124:34843,127.0.71.123:33471
--fault_before_start=1.0 with env {}
W20260929 01:58:07.285422 5399 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:07.285935 5399 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:07.286123 5399 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:58:07.286237 5399 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:07.299583 5399 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:07.300043 5399 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:58:07.312798 5399 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:41741,127.0.71.85:40765,127.0.71.86:42285,127.0.71.87:45011,127.0.71.88:39997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:42311
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=35021
--tserver_master_addrs=127.0.71.126:43673,127.0.71.125:44145,127.0.71.124:34843,127.0.71.123:33471
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:07.315212 5399 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:07.317855 5399 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:07.333009 5404 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:07.333165 5405 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:07.333305 5407 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:07.334589 5399 server_base.cc:1061] running on GCE node
I20260929 01:58:07.335609 5399 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:07.337697 5399 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:07.341388 5399 hybrid_clock.cc:648] HybridClock initialized: now 1790647087341174 us; error 90 us; skew 500 ppm
I20260929 01:58:07.342012 5399 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:58:07.356052 5399 webserver.cc:533] Webserver started at http://127.0.71.65:35021/ using document root <none> and password file <none>
I20260929 01:58:07.357307 5399 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:07.357520 5399 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:07.364941 5399 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 01:58:07.369218 5413 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:07.371284 5399 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:58:07.371577 5399 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "7bdce4d380404d9ca379b1bb9a64a29d"
format_stamp: "Formatted at 2026-09-29 01:58:02 on dist-test-slave-vz23"
I20260929 01:58:07.372740 5399 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:07.397351 5399 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:07.398577 5399 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:07.398948 5399 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:07.401137 5399 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:58:07.404772 5399 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:58:07.405014 5399 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:07.405207 5399 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:58:07.405324 5399 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:58:07.405923 5399 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:58:07.406076 5399 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:58:07.406221 5399 fault_injection.cc:59] injecting fault for kudu (pid 5399): FLAGS_fault_before_start (process will exit)
I20260929 01:58:07.430343 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:42311
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:43673,127.0.71.125:44145,127.0.71.124:34843
--webserver_port=35021
--webserver_interface=127.0.71.65
--builtin_ntp_servers=127.0.71.84:41741,127.0.71.85:40765,127.0.71.86:42285,127.0.71.87:45011,127.0.71.88:39997
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:43673,127.0.71.125:44145,127.0.71.124:34843,127.0.71.123:33471 with env {}
W20260929 01:58:07.889856 5421 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:07.890329 5421 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:07.890542 5421 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:07.900835 5421 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:07.901084 5421 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:58:07.914674 5421 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:41741,127.0.71.85:40765,127.0.71.86:42285,127.0.71.87:45011,127.0.71.88:39997
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:42311
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=35021
--tserver_master_addrs=127.0.71.126:43673,127.0.71.125:44145,127.0.71.124:34843,127.0.71.123:33471
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:07.917876 5421 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:07.920429 5421 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:07.935297 5429 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:07.935200 5426 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:07.935214 5427 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:07.937897 5421 server_base.cc:1061] running on GCE node
I20260929 01:58:07.938948 5421 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:07.940704 5421 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:07.944221 5421 hybrid_clock.cc:648] HybridClock initialized: now 1790647087943991 us; error 150 us; skew 500 ppm
I20260929 01:58:07.944829 5421 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:58:07.948643 5421 webserver.cc:533] Webserver started at http://127.0.71.65:35021/ using document root <none> and password file <none>
I20260929 01:58:07.949712 5421 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:07.949922 5421 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:07.957535 5421 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 01:58:07.962287 5435 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:07.964726 5421 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.004s
I20260929 01:58:07.965073 5421 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "7bdce4d380404d9ca379b1bb9a64a29d"
format_stamp: "Formatted at 2026-09-29 01:58:02 on dist-test-slave-vz23"
I20260929 01:58:07.966106 5421 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:07.992416 5421 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:07.993669 5421 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:07.993993 5421 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:07.996351 5421 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:58:07.999387 5421 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:58:07.999524 5421 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:07.999637 5421 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:58:07.999675 5421 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:08.050674 5421 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:42311
I20260929 01:58:08.050765 5547 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:42311 every 8 connection(s)
I20260929 01:58:08.052951 5421 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:58:08.063804 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 5421
I20260929 01:58:08.064272 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 5421
I20260929 01:58:08.103230 5548 heartbeater.cc:344] Connected to a master server at 127.0.71.124:34843
I20260929 01:58:08.104004 5550 heartbeater.cc:344] Connected to a master server at 127.0.71.123:33471
I20260929 01:58:08.104076 5549 heartbeater.cc:344] Connected to a master server at 127.0.71.126:43673
I20260929 01:58:08.103999 5548 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:08.104279 5550 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:08.104285 5549 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:08.105667 5550 heartbeater.cc:507] Master 127.0.71.123:33471 requested a full tablet report, sending...
I20260929 01:58:08.105747 5548 heartbeater.cc:507] Master 127.0.71.124:34843 requested a full tablet report, sending...
I20260929 01:58:08.108209 4959 ts_manager.cc:194] Re-registered known tserver with Master: 7bdce4d380404d9ca379b1bb9a64a29d (127.0.71.65:42311)
I20260929 01:58:08.108367 4583 ts_manager.cc:194] Re-registered known tserver with Master: 7bdce4d380404d9ca379b1bb9a64a29d (127.0.71.65:42311)
I20260929 01:58:08.108628 5549 heartbeater.cc:507] Master 127.0.71.126:43673 requested a full tablet report, sending...
I20260929 01:58:08.109556 4583 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:45135
I20260929 01:58:08.111018 5551 heartbeater.cc:344] Connected to a master server at 127.0.71.125:44145
I20260929 01:58:08.111279 5551 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:08.111251 5192 ts_manager.cc:194] Re-registered known tserver with Master: 7bdce4d380404d9ca379b1bb9a64a29d (127.0.71.65:42311)
I20260929 01:58:08.112032 5551 heartbeater.cc:507] Master 127.0.71.125:44145 requested a full tablet report, sending...
I20260929 01:58:08.113610 4511 ts_manager.cc:194] Re-registered known tserver with Master: 7bdce4d380404d9ca379b1bb9a64a29d (127.0.71.65:42311)
I20260929 01:58:08.113857 5543 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:58:08.211499 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 5421
I20260929 01:58:08.231498 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 5159
I20260929 01:58:08.239701 5222 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:58:08.365564 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 5159
I20260929 01:58:08.391726 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 4477
I20260929 01:58:08.393443 4541 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:58:08.527552 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 4477
I20260929 01:58:08.552569 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 4548
I20260929 01:58:08.554219 4613 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:58:08.684257 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 4548
I20260929 01:58:08.705493 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 4818
I20260929 01:58:08.707099 4989 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:58:08.835103 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 4818
Shutting down hive metastore.
2026-09-29T01:58:09Z chronyd exiting
2026-09-29T01:58:09Z chronyd exiting
2026-09-29T01:58:09Z chronyd exiting
2026-09-29T01:58:09Z chronyd exiting
2026-09-29T01:58:09Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestAddMaster/5 (31392 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/6
Loading random data
Initializing database '/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:58:09 dist-test-slave-vz23 krb5kdc[5575](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:58:09 dist-test-slave-vz23 krb5kdc[5575](info): set up 2 sockets
Sep 29 01:58:09 dist-test-slave-vz23 krb5kdc[5575](info): commencing operation
krb5kdc: starting...
W20260929 01:58:11.550005 285 mini_kdc.cc:121] Time spent starting KDC: real 2.194s user 0.004s sys 0.032s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 01:58:11 dist-test-slave-vz23 krb5kdc[5575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:58:11.901196 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:33965
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:33965,127.0.71.125:32867,127.0.71.124:43719 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:58:12.355554 5589 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:12.356045 5589 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:12.356251 5589 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:12.367108 5589 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:58:12.367298 5589 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:12.367453 5589 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:58:12.367559 5589 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:58:12.381042 5589 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:33965,127.0.71.125:32867,127.0.71.124:43719
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:33965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:12.383134 5589 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:12.384959 5589 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:12.396148 5594 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:12.396130 5595 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:12.397147 5597 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:12.397716 5589 server_base.cc:1061] running on GCE node
I20260929 01:58:12.398721 5589 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:58:12.398900 5589 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:58:12.398983 5589 hybrid_clock.cc:648] HybridClock initialized: now 1790647092398962 us; error 0 us; skew 500 ppm
I20260929 01:58:12.399336 5589 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:12 dist-test-slave-vz23 krb5kdc[5575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:12.403716 5589 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:58:12.406657 5589 webserver.cc:533] Webserver started at http://127.0.71.126:37133/ using document root <none> and password file <none>
I20260929 01:58:12.407630 5589 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:12.407794 5589 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:12.408210 5589 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:58:12.411513 5589 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "b0a89033ea9c493dbb253bd81a289e1c"
format_stamp: "Formatted at 2026-09-29 01:58:12 on dist-test-slave-vz23"
I20260929 01:58:12.412715 5589 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "b0a89033ea9c493dbb253bd81a289e1c"
format_stamp: "Formatted at 2026-09-29 01:58:12 on dist-test-slave-vz23"
I20260929 01:58:12.420262 5589 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260929 01:58:12.425631 5603 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:12.427764 5589 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.002s
I20260929 01:58:12.428061 5589 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "b0a89033ea9c493dbb253bd81a289e1c"
format_stamp: "Formatted at 2026-09-29 01:58:12 on dist-test-slave-vz23"
I20260929 01:58:12.428315 5589 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:12.453460 5589 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:12.459465 5589 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:12.459928 5589 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:12.487265 5589 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:33965
I20260929 01:58:12.487265 5654 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:33965 every 8 connection(s)
I20260929 01:58:12.489280 5589 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:58:12.494748 5655 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:58:12.497362 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 5589
I20260929 01:58:12.497617 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:58:12.502681 5655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:58:12 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.125@KRBTEST.COM" created.
I20260929 01:58:12.534554 5655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:58:12.536311 5604 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.125:32867: connect: Connection refused (error 111)
W20260929 01:58:12.538833 5655 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:32867: Network error: Client connection negotiation failed: client connection to 127.0.71.125:32867: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:58:12.588408 5655 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } attempt: 1
W20260929 01:58:12.591281 5655 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:32867: Network error: Client connection negotiation failed: client connection to 127.0.71.125:32867: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.0.71.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:58:12.640854 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:32867
--webserver_interface=127.0.71.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:33965,127.0.71.125:32867,127.0.71.124:43719 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:58:12.659714 5655 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } attempt: 2
W20260929 01:58:12.662724 5655 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:32867: Network error: Client connection negotiation failed: client connection to 127.0.71.125:32867: connect: Connection refused (error 111)
I20260929 01:58:12.754158 5655 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } attempt: 3
W20260929 01:58:12.756876 5655 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:32867: Network error: Client connection negotiation failed: client connection to 127.0.71.125:32867: connect: Connection refused (error 111)
I20260929 01:58:12.900398 5655 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } attempt: 4
W20260929 01:58:12.902827 5655 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:32867: Network error: Client connection negotiation failed: client connection to 127.0.71.125:32867: connect: Connection refused (error 111)
W20260929 01:58:13.065608 5664 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:13.066124 5664 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:13.066237 5664 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:13.075925 5664 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:58:13.076165 5664 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:13.076243 5664 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:58:13.076295 5664 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:58:13.088920 5664 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:33965,127.0.71.125:32867,127.0.71.124:43719
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:32867
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:13.090896 5664 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:13.092619 5664 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:13.104358 5670 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:13.104877 5669 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:13.105521 5672 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:13.107424 5664 server_base.cc:1061] running on GCE node
I20260929 01:58:13.108561 5664 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:58:13.108791 5664 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:58:13.108913 5664 hybrid_clock.cc:648] HybridClock initialized: now 1790647093108882 us; error 0 us; skew 500 ppm
I20260929 01:58:13.109333 5664 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:13 dist-test-slave-vz23 krb5kdc[5575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:13.114369 5664 init.cc:377] Logged in from keytab as kudu/127.0.71.125@KRBTEST.COM (short username kudu)
I20260929 01:58:13.117336 5664 webserver.cc:533] Webserver started at http://127.0.71.125:41803/ using document root <none> and password file <none>
I20260929 01:58:13.118309 5664 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:13.118448 5664 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:13.118911 5664 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:58:13.122074 5664 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-1/data/instance:
uuid: "9d0a5a9a6599458a88169f472ea39b10"
format_stamp: "Formatted at 2026-09-29 01:58:13 on dist-test-slave-vz23"
I20260929 01:58:13.123117 5664 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-1/wal/instance:
uuid: "9d0a5a9a6599458a88169f472ea39b10"
format_stamp: "Formatted at 2026-09-29 01:58:13 on dist-test-slave-vz23"
I20260929 01:58:13.130319 5664 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260929 01:58:13.135099 5678 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:13.136989 5664 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 01:58:13.137266 5664 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "9d0a5a9a6599458a88169f472ea39b10"
format_stamp: "Formatted at 2026-09-29 01:58:13 on dist-test-slave-vz23"
I20260929 01:58:13.137491 5664 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:13.169443 5664 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:13.182358 5664 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:13.182749 5664 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:13.202086 5655 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } attempt: 5
W20260929 01:58:13.204993 5655 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:32867: Network error: Client connection negotiation failed: client connection to 127.0.71.125:32867: connect: Connection refused (error 111)
I20260929 01:58:13.209400 5664 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:32867
I20260929 01:58:13.209400 5729 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:32867 every 8 connection(s)
I20260929 01:58:13.211227 5664 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:58:13.216912 5730 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:58:13.220203 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 5664
I20260929 01:58:13.220427 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:58:13.224864 5730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:58:13 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.124@KRBTEST.COM; defaulting to no policy
I20260929 01:58:13.255550 5730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.124@KRBTEST.COM" created.
Sep 29 01:58:13 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:58:13.278406 5730 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:58:13.280122 5682 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43719: connect: Connection refused (error 111)
W20260929 01:58:13.282421 5730 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43719: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43719: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.124@KRBTEST.COM" created.
I20260929 01:58:13.331948 5730 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } attempt: 1
W20260929 01:58:13.334802 5730 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43719: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43719: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:58:13.370601 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:43719
--webserver_interface=127.0.71.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:33965,127.0.71.125:32867,127.0.71.124:43719 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:58:13.403169 5730 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } attempt: 2
W20260929 01:58:13.405654 5730 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43719: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43719: connect: Connection refused (error 111)
I20260929 01:58:13.496935 5730 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } attempt: 3
W20260929 01:58:13.499758 5730 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43719: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43719: connect: Connection refused (error 111)
I20260929 01:58:13.643115 5730 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } attempt: 4
W20260929 01:58:13.646404 5730 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43719: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43719: connect: Connection refused (error 111)
I20260929 01:58:13.752388 5655 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } attempt: 6
Sep 29 01:58:13 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:58:13.772050 5655 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:58:13.776093 5655 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43719: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43719: connect: Connection refused (error 111)
W20260929 01:58:13.786116 5740 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:13.786526 5740 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:13.786716 5740 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:13.796911 5740 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:58:13.797056 5740 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:13.797171 5740 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:58:13.797259 5740 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:58:13.810197 5740 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:33965,127.0.71.125:32867,127.0.71.124:43719
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:43719
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:13.812232 5740 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:13.813982 5740 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:13.824293 5748 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:13.824215 5747 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:13.825505 5750 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:13.826068 5740 server_base.cc:1061] running on GCE node
I20260929 01:58:13.827108 5740 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:58:13.827266 5740 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:58:13.827347 5740 hybrid_clock.cc:648] HybridClock initialized: now 1790647093827320 us; error 0 us; skew 500 ppm
I20260929 01:58:13.827695 5740 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 01:58:13.828392 5655 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } attempt: 1
W20260929 01:58:13.831185 5655 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43719: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43719: connect: Connection refused (error 111)
Sep 29 01:58:13 dist-test-slave-vz23 krb5kdc[5575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:13.833053 5740 init.cc:377] Logged in from keytab as kudu/127.0.71.124@KRBTEST.COM (short username kudu)
I20260929 01:58:13.835628 5740 webserver.cc:533] Webserver started at http://127.0.71.124:46665/ using document root <none> and password file <none>
I20260929 01:58:13.836555 5740 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:13.836692 5740 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:13.837069 5740 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:58:13.840020 5740 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-2/data/instance:
uuid: "fd42a74e2edd48059be412278430c837"
format_stamp: "Formatted at 2026-09-29 01:58:13 on dist-test-slave-vz23"
I20260929 01:58:13.841059 5740 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-2/wal/instance:
uuid: "fd42a74e2edd48059be412278430c837"
format_stamp: "Formatted at 2026-09-29 01:58:13 on dist-test-slave-vz23"
I20260929 01:58:13.848397 5740 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.001s
I20260929 01:58:13.853334 5756 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:13.855388 5740 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.003s
I20260929 01:58:13.855659 5740 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "fd42a74e2edd48059be412278430c837"
format_stamp: "Formatted at 2026-09-29 01:58:13 on dist-test-slave-vz23"
I20260929 01:58:13.855876 5740 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:13.880226 5740 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:13.886013 5740 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:13.886512 5740 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:13.905467 5655 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } attempt: 2
W20260929 01:58:13.908110 5655 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43719: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43719: connect: Connection refused (error 111)
I20260929 01:58:13.914753 5740 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:43719
I20260929 01:58:13.914822 5807 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:43719 every 8 connection(s)
I20260929 01:58:13.916931 5740 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:58:13.923954 5808 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:58:13.925740 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 5740
I20260929 01:58:13.926113 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:58:13.934089 5808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:58:13 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:58:13.945737 5730 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } attempt: 5
Sep 29 01:58:13 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:58:13 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:58:13 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:58:13.969228 5808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:58:13.979157 5730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10: Bootstrap starting.
Sep 29 01:58:13 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:58:13.984848 5730 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10: Neither blocks nor log segments found. Creating new log.
Sep 29 01:58:13 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:58:13.986203 5730 log.cc:826] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10: Log is configured to *not* fsync() on all Append() calls
I20260929 01:58:13.989857 5730 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10: No bootstrap required, opened a new log
I20260929 01:58:13.992411 5808 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:58:13.998813 5730 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } }
I20260929 01:58:13.999346 5730 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:58:13.999547 5730 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9d0a5a9a6599458a88169f472ea39b10, State: Initialized, Role: FOLLOWER
I20260929 01:58:14.000571 5730 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } }
I20260929 01:58:14.001822 5818 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } }
I20260929 01:58:14.002153 5818 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: This master's current role is: FOLLOWER
Sep 29 01:58:13 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:58:14.003679 5730 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:58:14.021775 5829 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:58:14.022101 5829 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:58:14.022001 5655 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } attempt: 3
I20260929 01:58:14.015040 5664 master_runner.cc:187] Error getting master registration for 127.0.71.126:33965: OK, instance_id { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" instance_seqno: 1790647092479998 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:58:14.024058 5664 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:58:14.024665 5818 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:58:14.025025 5818 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } }
I20260929 01:58:14.027335 5818 leader_election.cc:290] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b0a89033ea9c493dbb253bd81a289e1c (127.0.71.126:33965), fd42a74e2edd48059be412278430c837 (127.0.71.124:43719)
I20260929 01:58:14.029696 5808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837: Bootstrap starting.
I20260929 01:58:14.035641 5808 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837: Neither blocks nor log segments found. Creating new log.
Sep 29 01:58:14 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647092, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:58:14.039281 5808 log.cc:826] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837: Log is configured to *not* fsync() on all Append() calls
I20260929 01:58:14.053102 5808 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837: No bootstrap required, opened a new log
I20260929 01:58:14.056644 5655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c: Bootstrap starting.
I20260929 01:58:14.062223 5655 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c: Neither blocks nor log segments found. Creating new log.
I20260929 01:58:14.062999 5808 raft_consensus.cc:359] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } }
I20260929 01:58:14.063488 5808 raft_consensus.cc:385] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:58:14.063648 5808 raft_consensus.cc:740] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: fd42a74e2edd48059be412278430c837, State: Initialized, Role: FOLLOWER
I20260929 01:58:14.064570 5655 log.cc:826] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c: Log is configured to *not* fsync() on all Append() calls
I20260929 01:58:14.064639 5808 consensus_queue.cc:260] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } }
I20260929 01:58:14.073638 5832 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } }
I20260929 01:58:14.074515 5832 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:14.075145 5808 sys_catalog.cc:565] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:58:14.075208 5783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d0a5a9a6599458a88169f472ea39b10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd42a74e2edd48059be412278430c837" is_pre_election: true
I20260929 01:58:14.075889 5783 raft_consensus.cc:2468] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d0a5a9a6599458a88169f472ea39b10 in term 0.
I20260929 01:58:14.076045 5655 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c: No bootstrap required, opened a new log
I20260929 01:58:14.076917 5682 leader_election.cc:304] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d0a5a9a6599458a88169f472ea39b10, fd42a74e2edd48059be412278430c837; no voters:
I20260929 01:58:14.077925 5818 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:58:14.078148 5818 raft_consensus.cc:493] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:58:14.078284 5818 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:58:14.082412 5818 raft_consensus.cc:515] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } }
I20260929 01:58:14.084872 5783 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d0a5a9a6599458a88169f472ea39b10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "fd42a74e2edd48059be412278430c837"
I20260929 01:58:14.085273 5783 raft_consensus.cc:3060] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:58:14.089668 5783 raft_consensus.cc:2468] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a5a9a6599458a88169f472ea39b10 in term 1.
I20260929 01:58:14.090219 5818 leader_election.cc:290] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [CANDIDATE]: Term 1 election: Requested vote from peers b0a89033ea9c493dbb253bd81a289e1c (127.0.71.126:33965), fd42a74e2edd48059be412278430c837 (127.0.71.124:43719)
I20260929 01:58:14.090660 5682 leader_election.cc:304] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 9d0a5a9a6599458a88169f472ea39b10, fd42a74e2edd48059be412278430c837; no voters:
I20260929 01:58:14.091058 5818 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:58:14.096589 5818 raft_consensus.cc:697] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [term 1 LEADER]: Becoming Leader. State: Replica: 9d0a5a9a6599458a88169f472ea39b10, State: Running, Role: LEADER
I20260929 01:58:14.097430 5818 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } }
I20260929 01:58:14.103852 5835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d0a5a9a6599458a88169f472ea39b10. Latest consensus state: current_term: 1 leader_uuid: "9d0a5a9a6599458a88169f472ea39b10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } }
I20260929 01:58:14.104272 5835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:14.105644 5655 raft_consensus.cc:359] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } }
I20260929 01:58:14.106150 5655 raft_consensus.cc:385] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:58:14.106367 5655 raft_consensus.cc:740] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a89033ea9c493dbb253bd81a289e1c, State: Initialized, Role: FOLLOWER
I20260929 01:58:14.107478 5655 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } }
I20260929 01:58:14.115214 5837 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:58:14.121824 5838 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } }
I20260929 01:58:14.123104 5838 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:14.123453 5655 sys_catalog.cc:565] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:58:14.123564 5630 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d0a5a9a6599458a88169f472ea39b10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a89033ea9c493dbb253bd81a289e1c" is_pre_election: true
I20260929 01:58:14.124545 5630 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 9d0a5a9a6599458a88169f472ea39b10 in term 0.
I20260929 01:58:14.126662 5629 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "9d0a5a9a6599458a88169f472ea39b10" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b0a89033ea9c493dbb253bd81a289e1c"
I20260929 01:58:14.127233 5629 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:58:14.131206 5837 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:58:14.133384 5629 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 9d0a5a9a6599458a88169f472ea39b10 in term 1.
W20260929 01:58:14.139530 5762 tablet.cc:2433] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:58:14.154119 5783 raft_consensus.cc:1275] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837 [term 1 FOLLOWER]: Refusing update from remote peer 9d0a5a9a6599458a88169f472ea39b10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:58:14.156240 5835 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:58:14.170306 5629 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [term 1 FOLLOWER]: Refusing update from remote peer 9d0a5a9a6599458a88169f472ea39b10: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:58:14.171815 5835 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:58:14.185708 5832 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d0a5a9a6599458a88169f472ea39b10. Latest consensus state: current_term: 1 leader_uuid: "9d0a5a9a6599458a88169f472ea39b10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } }
I20260929 01:58:14.186012 5832 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:58:14.186657 5684 tablet.cc:2433] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:58:14.187469 5844 mvcc.cc:204] Tried to move back new op lower bound from 7334490497636454400 to 7334490497430032384. Current Snapshot: MvccSnapshot[applied={T|T < 7334490497636454400}]
I20260929 01:58:14.190990 5857 mvcc.cc:204] Tried to move back new op lower bound from 7334490497636454400 to 7334490497430032384. Current Snapshot: MvccSnapshot[applied={T|T < 7334490497636454400}]
I20260929 01:58:14.193161 5835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d0a5a9a6599458a88169f472ea39b10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } }
I20260929 01:58:14.193711 5835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:14.202066 5832 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d0a5a9a6599458a88169f472ea39b10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } }
I20260929 01:58:14.202502 5832 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:14.205497 5837 catalog_manager.cc:1396] Generated new cluster ID: 3cf4c66a24154d159ba4bf1f8d3fc7c2
I20260929 01:58:14.205840 5837 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:58:14.208681 5838 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d0a5a9a6599458a88169f472ea39b10. Latest consensus state: current_term: 1 leader_uuid: "9d0a5a9a6599458a88169f472ea39b10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } }
I20260929 01:58:14.209071 5838 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:14.212651 5863 catalog_manager.cc:818] Waiting for catalog manager background task thread to start: Service unavailable: Catalog manager is not initialized. State: Starting
W20260929 01:58:14.214289 5609 tablet.cc:2433] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:58:14.226059 5835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d0a5a9a6599458a88169f472ea39b10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } }
I20260929 01:58:14.226446 5835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:14.230994 5837 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 01:58:14.232757 5838 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d0a5a9a6599458a88169f472ea39b10" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } }
I20260929 01:58:14.233047 5838 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:14.233613 5837 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:58:14.263628 5837 catalog_manager.cc:6219] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10: Generated new TSK 0
I20260929 01:58:14.265323 5837 catalog_manager.cc:1563] Initializing in-progress tserver states...
W20260929 01:58:14.278177 5866 catalog_manager.cc:1607] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:58:14.278399 5866 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
W20260929 01:58:14.492368 285 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.0.71.65: real 0.102s user 0.001s sys 0.027s
I20260929 01:58:14.493804 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:33965,127.0.71.125:32867,127.0.71.124:43719
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:58:15.046452 5874 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:15.046819 5874 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:15.046986 5874 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:15.057387 5874 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:15.057717 5874 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:58:15.071518 5874 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:33965,127.0.71.125:32867,127.0.71.124:43719
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:15.074028 5874 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:15.076045 5874 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:15.090472 5882 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:15.090224 5880 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:15.090591 5879 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:15.093881 5874 server_base.cc:1061] running on GCE node
I20260929 01:58:15.094980 5874 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:58:15.095244 5874 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:58:15.095371 5874 hybrid_clock.cc:648] HybridClock initialized: now 1790647095095326 us; error 0 us; skew 500 ppm
I20260929 01:58:15.095804 5874 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:15 dist-test-slave-vz23 krb5kdc[5575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:15.102077 5874 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:58:15.105089 5874 webserver.cc:533] Webserver started at http://127.0.71.65:39267/ using document root <none> and password file <none>
I20260929 01:58:15.106240 5874 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:15.106453 5874 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:15.106922 5874 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:58:15.110916 5874 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "22d7eb8342a54591af883185205b9903"
format_stamp: "Formatted at 2026-09-29 01:58:15 on dist-test-slave-vz23"
I20260929 01:58:15.112082 5874 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "22d7eb8342a54591af883185205b9903"
format_stamp: "Formatted at 2026-09-29 01:58:15 on dist-test-slave-vz23"
I20260929 01:58:15.120406 5874 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.003s sys 0.002s
I20260929 01:58:15.126807 5888 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:15.129124 5874 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.002s
I20260929 01:58:15.129419 5874 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "22d7eb8342a54591af883185205b9903"
format_stamp: "Formatted at 2026-09-29 01:58:15 on dist-test-slave-vz23"
I20260929 01:58:15.129700 5874 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:15.152102 5874 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:15.157986 5874 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:15.158377 5874 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:15.159667 5874 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:58:15.162541 5874 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:58:15.162739 5874 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:15.162922 5874 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:58:15.163003 5874 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:15.207166 5874 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:39891
I20260929 01:58:15.207197 6000 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:39891 every 8 connection(s)
I20260929 01:58:15.209271 5874 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:58:15.209980 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 5874
I20260929 01:58:15.210215 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
I20260929 01:58:15.233799 5863 catalog_manager.cc:1308] Loaded cluster ID: 3cf4c66a24154d159ba4bf1f8d3fc7c2
I20260929 01:58:15.234038 5863 catalog_manager.cc:1601] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837: loading cluster ID for follower catalog manager: success
I20260929 01:58:15.250039 5863 catalog_manager.cc:1623] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837: acquiring CA information for follower catalog manager: success
Sep 29 01:58:15 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:58:15.259747 5863 catalog_manager.cc:1651] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:58:15.272822 6001 heartbeater.cc:344] Connected to a master server at 127.0.71.124:43719
I20260929 01:58:15.273347 6001 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:15.274919 6001 heartbeater.cc:507] Master 127.0.71.124:43719 requested a full tablet report, sending...
Sep 29 01:58:15 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:58:15.278767 5773 ts_manager.cc:194] Registered new tserver with Master: 22d7eb8342a54591af883185205b9903 (127.0.71.65:39891)
I20260929 01:58:15.280759 5866 catalog_manager.cc:1308] Loaded cluster ID: 3cf4c66a24154d159ba4bf1f8d3fc7c2
I20260929 01:58:15.280884 5866 catalog_manager.cc:1601] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c: loading cluster ID for follower catalog manager: success
I20260929 01:58:15.286486 5866 catalog_manager.cc:1623] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c: acquiring CA information for follower catalog manager: success
I20260929 01:58:15.292083 5866 catalog_manager.cc:1651] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Sep 29 01:58:15 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:58:15.296123 6003 heartbeater.cc:344] Connected to a master server at 127.0.71.125:32867
I20260929 01:58:15.296375 6003 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:15.297053 6003 heartbeater.cc:507] Master 127.0.71.125:32867 requested a full tablet report, sending...
I20260929 01:58:15.299494 5694 ts_manager.cc:194] Registered new tserver with Master: 22d7eb8342a54591af883185205b9903 (127.0.71.65:39891)
I20260929 01:58:15.301988 5694 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:41349
I20260929 01:58:15.302922 6002 heartbeater.cc:344] Connected to a master server at 127.0.71.126:33965
I20260929 01:58:15.303136 6002 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:15.303860 6002 heartbeater.cc:507] Master 127.0.71.126:33965 requested a full tablet report, sending...
I20260929 01:58:15.306414 5619 ts_manager.cc:194] Registered new tserver with Master: 22d7eb8342a54591af883185205b9903 (127.0.71.65:39891)
I20260929 01:58:15.324100 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
WARNING: no policy specified for kudu/127.0.71.123@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.123@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.123 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.123 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.123@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.123@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.123 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.123 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:58:15.494032 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.123:45007
--webserver_interface=127.0.71.123
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.123
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:33965,127.0.71.125:32867,127.0.71.124:43719,127.0.71.123:45007 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:58:15.915866 6016 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:15.916268 6016 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:15.916621 6016 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:15.927212 6016 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:58:15.927372 6016 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:15.927480 6016 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:58:15.927570 6016 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:58:15.940403 6016 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:33965,127.0.71.125:32867,127.0.71.124:43719,127.0.71.123:45007
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.123
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.123:45007
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.0.71.123
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:15.942551 6016 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:15.944391 6016 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:15.955084 6022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:15.955117 6021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:15.956028 6024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:15.956807 6016 server_base.cc:1061] running on GCE node
I20260929 01:58:15.957858 6016 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:58:15.958078 6016 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:58:15.958181 6016 hybrid_clock.cc:648] HybridClock initialized: now 1790647095958160 us; error 0 us; skew 500 ppm
I20260929 01:58:15.958621 6016 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:15 dist-test-slave-vz23 krb5kdc[5575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:15.963419 6016 init.cc:377] Logged in from keytab as kudu/127.0.71.123@KRBTEST.COM (short username kudu)
I20260929 01:58:15.966079 6016 webserver.cc:533] Webserver started at http://127.0.71.123:36217/ using document root <none> and password file <none>
I20260929 01:58:15.967088 6016 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:15.967216 6016 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:15.967684 6016 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:58:15.970757 6016 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-3/data/instance:
uuid: "9ece97800ec6402a8930e980360d77a1"
format_stamp: "Formatted at 2026-09-29 01:58:15 on dist-test-slave-vz23"
I20260929 01:58:15.971763 6016 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-3/wal/instance:
uuid: "9ece97800ec6402a8930e980360d77a1"
format_stamp: "Formatted at 2026-09-29 01:58:15 on dist-test-slave-vz23"
I20260929 01:58:15.979336 6016 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260929 01:58:15.984724 6030 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:15.986975 6016 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.004s sys 0.000s
I20260929 01:58:15.987309 6016 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "9ece97800ec6402a8930e980360d77a1"
format_stamp: "Formatted at 2026-09-29 01:58:15 on dist-test-slave-vz23"
I20260929 01:58:15.987600 6016 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:16.013993 6016 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:16.021878 6016 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:16.022285 6016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:16.049433 6016 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.123:45007
I20260929 01:58:16.049511 6081 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.123:45007 every 8 connection(s)
I20260929 01:58:16.051587 6016 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
I20260929 01:58:16.057389 6082 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:58:16.059231 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 6016
I20260929 01:58:16.068379 6082 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:58:16 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
Sep 29 01:58:16 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:58:16.093091 6082 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:58:16 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:58:16.110874 6082 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:58:16 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:58:16.126885 6082 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.123" port: 45007 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:58:16 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647095, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
I20260929 01:58:16.146953 6082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1: Bootstrap starting.
I20260929 01:58:16.151214 6082 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1: Neither blocks nor log segments found. Creating new log.
I20260929 01:58:16.152738 6082 log.cc:826] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1: Log is configured to *not* fsync() on all Append() calls
I20260929 01:58:16.155910 6082 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1: No bootstrap required, opened a new log
I20260929 01:58:16.161958 6082 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 45007 } }
I20260929 01:58:16.162432 6082 raft_consensus.cc:385] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:58:16.162585 6082 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 9ece97800ec6402a8930e980360d77a1, State: Initialized, Role: FOLLOWER
I20260929 01:58:16.163312 6082 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 45007 } }
I20260929 01:58:16.164578 6091 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 45007 } } }
I20260929 01:58:16.164912 6091 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:16.165817 6082 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:58:16.178216 6102 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:58:16.178419 6102 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:58:16.189235 6016 master_runner.cc:429] Detected that this master 9ece97800ec6402a8930e980360d77a1 is joining an existing cluster
I20260929 01:58:16.189417 6016 catalog_manager.cc:1037] Shutting down the Catalog Manager
I20260929 01:58:16.190434 6016 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260929 01:58:16.190963 6016 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260929 01:58:16.191214 6016 tablet_replica.cc:333] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1: stopping tablet replica
I20260929 01:58:16.193650 6016 master_runner.cc:434] Initiating AddMaster RPC to add 127.0.71.123:45007
Sep 29 01:58:16 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647093, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
I20260929 01:58:16.260421 5694 catalog_manager.cc:7351] Initiating ChangeConfig request to add master 127.0.71.123:45007: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "9ece97800ec6402a8930e980360d77a1"
member_type: NON_VOTER
last_known_addr {
host: "127.0.71.123"
port: 45007
}
attrs {
promote: true
}
}
dest_uuid: "9d0a5a9a6599458a88169f472ea39b10"
cas_config_opid_index: -1
I20260929 01:58:16.262526 5694 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: true } }
I20260929 01:58:16.266471 5630 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [term 1 FOLLOWER]: Refusing update from remote peer 9d0a5a9a6599458a88169f472ea39b10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 01:58:16.267184 5782 raft_consensus.cc:1275] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837 [term 1 FOLLOWER]: Refusing update from remote peer 9d0a5a9a6599458a88169f472ea39b10: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 01:58:16.267671 5835 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
W20260929 01:58:16.268079 5682 consensus_peers.cc:597] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 -> Peer 9ece97800ec6402a8930e980360d77a1 (127.0.71.123:45007): Couldn't send request to peer 9ece97800ec6402a8930e980360d77a1. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260929 01:58:16.268501 5835 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:58:16.272342 6114 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9ece97800ec6402a8930e980360d77a1 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: true } } }
I20260929 01:58:16.273579 5630 raft_consensus.cc:2955] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9ece97800ec6402a8930e980360d77a1 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: true } } }
I20260929 01:58:16.274149 5782 raft_consensus.cc:2955] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 9ece97800ec6402a8930e980360d77a1 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: true } } }
I20260929 01:58:16.277789 6114 catalog_manager.cc:7355] Successfully completed master ChangeConfig request to add master 127.0.71.123:45007
I20260929 01:58:16.278890 6016 master.cc:562] Master@127.0.71.123:45007 shutting down...
I20260929 01:58:16.291179 5835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9d0a5a9a6599458a88169f472ea39b10" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: true } } }
I20260929 01:58:16.291599 5835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:16.298974 6118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d0a5a9a6599458a88169f472ea39b10" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: true } } }
I20260929 01:58:16.299467 6118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:16.299749 6118 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d0a5a9a6599458a88169f472ea39b10" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: true } } }
I20260929 01:58:16.299988 6118 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:16.308430 6003 heartbeater.cc:499] Master 127.0.71.125:32867 was elected leader, sending a full tablet report...
I20260929 01:58:16.308862 6117 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d0a5a9a6599458a88169f472ea39b10" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: true } } }
I20260929 01:58:16.309276 6117 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:16.310616 6119 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d0a5a9a6599458a88169f472ea39b10" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: true } } }
I20260929 01:58:16.311091 6119 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:58:16.328555 6034 connection.cc:466] server connection from 127.0.0.1:50920 torn down before Call kudu.master.MasterService.ListTables from 127.0.0.1:50920 (request call id 7) could send its response
I20260929 01:58:16.342006 6016 master.cc:584] Master@127.0.71.123:45007 shutdown complete.
I20260929 01:58:16.342175 6016 master_runner.cc:306] Clearing existing system tablet
I20260929 01:58:16.342600 6016 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:16.342733 6016 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:16.346083 6016 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 01:58:16.348064 6123 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:16.348553 6016 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:16.348927 6016 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260929 01:58:16.349026 6016 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "9ece97800ec6402a8930e980360d77a1"
format_stamp: "Formatted at 2026-09-29 01:58:15 on dist-test-slave-vz23"
I20260929 01:58:16.353228 6016 ts_tablet_manager.cc:1962] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:58:16.359047 6016 ts_tablet_manager.cc:1975] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 01:58:16.359295 6016 log.cc:1199] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260929 01:58:16.359790 6016 ts_tablet_manager.cc:1996] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1: Deleting consensus metadata
I20260929 01:58:16.360105 6016 master_runner.cc:316] Copying system tablet from 127.0.71.125:32867
I20260929 01:58:16.363648 6016 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1: tablet copy: Beginning tablet copy session from remote peer at address 127.0.71.125:32867
I20260929 01:58:16.381656 5715 tablet_copy_service.cc:140] P 9d0a5a9a6599458a88169f472ea39b10: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 9ece97800ec6402a8930e980360d77a1 ({username='kudu', principal='kudu/127.0.71.123@KRBTEST.COM'} at 127.0.0.1:35494)
I20260929 01:58:16.382094 5715 tablet_copy_service.cc:161] P 9d0a5a9a6599458a88169f472ea39b10: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 9ece97800ec6402a8930e980360d77a1 at {username='kudu', principal='kudu/127.0.71.123@KRBTEST.COM'} at 127.0.0.1:35494: session id = 9ece97800ec6402a8930e980360d77a1-00000000000000000000000000000000
I20260929 01:58:16.387116 5715 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 01:58:16.390275 6016 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 01:58:16.396572 6016 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1: tablet copy: Starting download of 0 data blocks...
I20260929 01:58:16.397002 6016 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1: tablet copy: Starting download of 1 WAL segments...
I20260929 01:58:16.400624 6016 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 01:58:16.404969 5715 tablet_copy_service.cc:342] P 9d0a5a9a6599458a88169f472ea39b10: Request end of tablet copy session 9ece97800ec6402a8930e980360d77a1-00000000000000000000000000000000 received from {username='kudu', principal='kudu/127.0.71.123@KRBTEST.COM'} at 127.0.0.1:35494
I20260929 01:58:16.405261 5715 tablet_copy_service.cc:434] P 9d0a5a9a6599458a88169f472ea39b10: ending tablet copy session 9ece97800ec6402a8930e980360d77a1-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 9ece97800ec6402a8930e980360d77a1
I20260929 01:58:16.417694 6016 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:16.421347 6131 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:16.421967 6130 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:16.423468 6133 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:16.423478 6016 server_base.cc:1061] running on GCE node
I20260929 01:58:16.424068 6016 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:58:16.424192 6016 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:58:16.424393 6016 hybrid_clock.cc:648] HybridClock initialized: now 1790647096424340 us; error 0 us; skew 500 ppm
I20260929 01:58:16.424687 6016 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:16 dist-test-slave-vz23 krb5kdc[5575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647096, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:16.427284 6016 init.cc:377] Logged in from keytab as kudu/127.0.71.123@KRBTEST.COM (short username kudu)
I20260929 01:58:16.428807 6016 webserver.cc:533] Webserver started at http://127.0.71.123:39451/ using document root <none> and password file <none>
I20260929 01:58:16.429211 6016 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:16.429332 6016 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:16.433118 6016 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.000s sys 0.003s
I20260929 01:58:16.435525 6139 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:16.436296 6016 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260929 01:58:16.436553 6016 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "9ece97800ec6402a8930e980360d77a1"
format_stamp: "Formatted at 2026-09-29 01:58:15 on dist-test-slave-vz23"
I20260929 01:58:16.436762 6016 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:16.458276 6016 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:16.462864 6016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:16.486766 6016 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.123:45007
I20260929 01:58:16.486865 6190 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.123:45007 every 8 connection(s)
I20260929 01:58:16.488207 6016 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
I20260929 01:58:16.495447 6191 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:58:16.500171 6191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1: Bootstrap starting.
I20260929 01:58:16.520520 6191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:58:16.521193 6191 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1: Bootstrap complete.
I20260929 01:58:16.522449 6191 raft_consensus.cc:359] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: true } }
I20260929 01:58:16.522779 6191 raft_consensus.cc:740] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 9ece97800ec6402a8930e980360d77a1, State: Initialized, Role: LEARNER
I20260929 01:58:16.523121 6191 consensus_queue.cc:260] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: true } }
I20260929 01:58:16.524217 6194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: true } } }
I20260929 01:58:16.524626 6194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1 [sys.catalog]: This master's current role is: LEARNER
I20260929 01:58:16.524714 6191 sys_catalog.cc:565] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:58:16.536070 6205 catalog_manager.cc:1308] Loaded cluster ID: 3cf4c66a24154d159ba4bf1f8d3fc7c2
I20260929 01:58:16.536249 6205 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1: loading cluster ID for follower catalog manager: success
I20260929 01:58:16.539736 6205 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1: acquiring CA information for follower catalog manager: success
I20260929 01:58:16.542425 6205 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:58:16.561204 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 5874
Sep 29 01:58:16 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647091, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 01:58:16.583572 5996 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:58:16.684386 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 5874
I20260929 01:58:16.707144 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:39891
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:33965,127.0.71.125:32867,127.0.71.124:43719
--webserver_port=39267
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--tserver_master_addrs=127.0.71.126:33965,127.0.71.125:32867,127.0.71.124:43719,127.0.71.123:45007 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:58:16.713495 6166 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260929 01:58:16.714485 6194 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d0a5a9a6599458a88169f472ea39b10. Latest consensus state: current_term: 1 leader_uuid: "9d0a5a9a6599458a88169f472ea39b10" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: true } } }
I20260929 01:58:16.714829 6194 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1 [sys.catalog]: This master's current role is: LEARNER
W20260929 01:58:17.156993 6209 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:17.157408 6209 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:17.157600 6209 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:17.167357 6209 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:17.167654 6209 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:58:17.180074 6209 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:39891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=39267
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:33965,127.0.71.125:32867,127.0.71.124:43719,127.0.71.123:45007
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:17.182328 6209 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:17.184576 6209 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:17.196633 6215 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:17.196661 6214 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:17.197358 6217 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:17.197697 6209 server_base.cc:1061] running on GCE node
I20260929 01:58:17.198417 6209 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:58:17.198650 6209 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:58:17.198767 6209 hybrid_clock.cc:648] HybridClock initialized: now 1790647097198741 us; error 0 us; skew 500 ppm
I20260929 01:58:17.199167 6209 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:17 dist-test-slave-vz23 krb5kdc[5575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:17.203445 6209 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:58:17.205967 6209 webserver.cc:533] Webserver started at http://127.0.71.65:39267/ using document root <none> and password file <none>
I20260929 01:58:17.207048 6209 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:17.207295 6209 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:17.207410 6114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d0a5a9a6599458a88169f472ea39b10" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: true } } }
I20260929 01:58:17.207680 6114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:17.209553 6223 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10: attempting to promote NON_VOTER 9ece97800ec6402a8930e980360d77a1 to VOTER
I20260929 01:58:17.210844 6223 consensus_queue.cc:237] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: false } }
I20260929 01:58:17.214286 6166 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1 [term 1 LEARNER]: Refusing update from remote peer 9d0a5a9a6599458a88169f472ea39b10: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:58:17.214543 5782 raft_consensus.cc:1275] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837 [term 1 FOLLOWER]: Refusing update from remote peer 9d0a5a9a6599458a88169f472ea39b10: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:58:17.215509 5630 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [term 1 FOLLOWER]: Refusing update from remote peer 9d0a5a9a6599458a88169f472ea39b10: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:58:17.216068 6222 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:58:17.217378 6222 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:58:17.218806 6222 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [LEADER]: Connected to new peer: Peer: permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:58:17.225560 5630 raft_consensus.cc:2955] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9ece97800ec6402a8930e980360d77a1 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: false } } }
I20260929 01:58:17.224375 6225 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9ece97800ec6402a8930e980360d77a1 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: false } } }
I20260929 01:58:17.237541 6223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "9d0a5a9a6599458a88169f472ea39b10" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: false } } }
I20260929 01:58:17.237864 6223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:17.238000 5782 raft_consensus.cc:2955] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9ece97800ec6402a8930e980360d77a1 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: false } } }
I20260929 01:58:17.238493 6223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d0a5a9a6599458a88169f472ea39b10" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: false } } }
I20260929 01:58:17.238706 6223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:17.239112 6223 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d0a5a9a6599458a88169f472ea39b10" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: false } } }
I20260929 01:58:17.239338 6223 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:17.243477 6114 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d0a5a9a6599458a88169f472ea39b10" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: false } } }
I20260929 01:58:17.243889 6114 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:17.245002 6165 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 9ece97800ec6402a8930e980360d77a1 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: false } } }
I20260929 01:58:17.242712 6228 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d0a5a9a6599458a88169f472ea39b10" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: false } } }
I20260929 01:58:17.254186 6228 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:17.255095 6229 sys_catalog.cc:455] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d0a5a9a6599458a88169f472ea39b10" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: false } } }
I20260929 01:58:17.255489 6229 sys_catalog.cc:458] T 00000000000000000000000000000000 P fd42a74e2edd48059be412278430c837 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:17.258234 6209 fs_manager.cc:714] Time spent opening directory manager: real 0.048s user 0.022s sys 0.024s
I20260929 01:58:17.256893 6230 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "9d0a5a9a6599458a88169f472ea39b10" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: false } } }
I20260929 01:58:17.259007 6230 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9ece97800ec6402a8930e980360d77a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:17.262519 6231 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:17.264869 6209 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260929 01:58:17.265218 6209 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "22d7eb8342a54591af883185205b9903"
format_stamp: "Formatted at 2026-09-29 01:58:15 on dist-test-slave-vz23"
I20260929 01:58:17.266232 6209 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:17.293254 6209 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:17.299932 6209 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:17.300289 6209 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:17.301863 6209 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:58:17.304059 6209 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:58:17.304184 6209 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:17.304272 6209 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:58:17.304337 6209 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:17.354589 6209 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:39891
I20260929 01:58:17.354728 6343 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:39891 every 8 connection(s)
I20260929 01:58:17.356755 6209 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:58:17.364401 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 6209
Sep 29 01:58:17 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:58:17 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:58:17 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:58:17 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647097, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
I20260929 01:58:17.418743 6345 heartbeater.cc:344] Connected to a master server at 127.0.71.124:43719
I20260929 01:58:17.418884 6346 heartbeater.cc:344] Connected to a master server at 127.0.71.126:33965
I20260929 01:58:17.419287 6345 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:17.419296 6346 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:17.420616 6344 heartbeater.cc:344] Connected to a master server at 127.0.71.123:45007
I20260929 01:58:17.420840 6346 heartbeater.cc:507] Master 127.0.71.126:33965 requested a full tablet report, sending...
I20260929 01:58:17.420852 6344 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:17.421505 6344 heartbeater.cc:507] Master 127.0.71.123:45007 requested a full tablet report, sending...
I20260929 01:58:17.422504 6347 heartbeater.cc:344] Connected to a master server at 127.0.71.125:32867
I20260929 01:58:17.422675 6345 heartbeater.cc:507] Master 127.0.71.124:43719 requested a full tablet report, sending...
I20260929 01:58:17.422719 6347 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:17.423403 6347 heartbeater.cc:507] Master 127.0.71.125:32867 requested a full tablet report, sending...
I20260929 01:58:17.424352 6156 ts_manager.cc:194] Registered new tserver with Master: 22d7eb8342a54591af883185205b9903 (127.0.71.65:39891)
I20260929 01:58:17.425556 5619 ts_manager.cc:194] Re-registered known tserver with Master: 22d7eb8342a54591af883185205b9903 (127.0.71.65:39891)
I20260929 01:58:17.425645 5694 ts_manager.cc:194] Re-registered known tserver with Master: 22d7eb8342a54591af883185205b9903 (127.0.71.65:39891)
I20260929 01:58:17.427197 5694 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:56607
I20260929 01:58:17.428344 5773 ts_manager.cc:194] Re-registered known tserver with Master: 22d7eb8342a54591af883185205b9903 (127.0.71.65:39891)
I20260929 01:58:17.429967 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:58:17.446789 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:58:17.447132 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 5589
I20260929 01:58:17.448941 5650 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:58:17.591928 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 5589
I20260929 01:58:17.618466 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:33965
--webserver_interface=127.0.71.126
--webserver_port=37133
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:33965,127.0.71.125:32867,127.0.71.124:43719
--master_addresses=127.0.71.126:33965,127.0.71.125:32867,127.0.71.124:43719,127.0.71.123:45007 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:58:17.673251 5681 consensus_peers.cc:597] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 -> Peer b0a89033ea9c493dbb253bd81a289e1c (127.0.71.126:33965): Couldn't send request to peer b0a89033ea9c493dbb253bd81a289e1c. Status: Network error: Client connection negotiation failed: client connection to 127.0.71.126:33965: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260929 01:58:18.077762 6358 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:18.078205 6358 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:18.078310 6358 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:18.089615 6358 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:58:18.089782 6358 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:18.089845 6358 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:58:18.089890 6358 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:58:18.102259 6358 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:33965,127.0.71.125:32867,127.0.71.124:43719,127.0.71.123:45007
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:33965
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=37133
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:18.104351 6358 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:18.106062 6225 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d0a5a9a6599458a88169f472ea39b10" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: false } } }
I20260929 01:58:18.106482 6358 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260929 01:58:18.106539 6225 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: This master's current role is: LEADER
W20260929 01:58:18.120363 6369 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:18.120653 6368 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:18.120915 6371 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:18.122370 6358 server_base.cc:1061] running on GCE node
I20260929 01:58:18.123500 6358 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:58:18.123706 6358 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:58:18.123780 6358 hybrid_clock.cc:648] HybridClock initialized: now 1790647098123758 us; error 0 us; skew 500 ppm
I20260929 01:58:18.124185 6358 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:18 dist-test-slave-vz23 krb5kdc[5575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:18.129638 6358 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:58:18.132190 6358 webserver.cc:533] Webserver started at http://127.0.71.126:37133/ using document root <none> and password file <none>
I20260929 01:58:18.133299 6358 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:18.133478 6358 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:18.140915 6358 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 01:58:18.145649 6377 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:18.148141 6358 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:58:18.148435 6358 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "b0a89033ea9c493dbb253bd81a289e1c"
format_stamp: "Formatted at 2026-09-29 01:58:12 on dist-test-slave-vz23"
I20260929 01:58:18.149576 6358 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:18.169968 6358 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:18.175496 6358 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:18.175860 6358 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:18.202868 6428 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:33965 every 8 connection(s)
I20260929 01:58:18.202890 6358 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:33965
I20260929 01:58:18.205410 6358 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:58:18.215179 6429 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:58:18.215763 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 6358
I20260929 01:58:18.216142 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 6209
I20260929 01:58:18.223268 6429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c: Bootstrap starting.
I20260929 01:58:18.250422 6429 log.cc:826] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c: Log is configured to *not* fsync() on all Append() calls
I20260929 01:58:18.257316 6339 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:58:18.268082 6429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:58:18.268872 6429 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c: Bootstrap complete.
I20260929 01:58:18.276922 6429 raft_consensus.cc:359] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: false } }
I20260929 01:58:18.278579 6429 raft_consensus.cc:740] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b0a89033ea9c493dbb253bd81a289e1c, State: Initialized, Role: FOLLOWER
I20260929 01:58:18.279811 6429 consensus_queue.cc:260] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: false } }
I20260929 01:58:18.281220 6435 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: false } } }
I20260929 01:58:18.281564 6435 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:18.282354 6429 sys_catalog.cc:565] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:58:18.296432 6446 catalog_manager.cc:1308] Loaded cluster ID: 3cf4c66a24154d159ba4bf1f8d3fc7c2
I20260929 01:58:18.296835 6446 catalog_manager.cc:1601] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c: loading cluster ID for follower catalog manager: success
I20260929 01:58:18.307700 6446 catalog_manager.cc:1623] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c: acquiring CA information for follower catalog manager: success
I20260929 01:58:18.312537 6446 catalog_manager.cc:1651] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Sep 29 01:58:18 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:58:18.384466 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 6209
I20260929 01:58:18.406812 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:39891
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:33965,127.0.71.125:32867,127.0.71.124:43719
--webserver_port=39267
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--tserver_master_addrs=127.0.71.126:33965,127.0.71.125:32867,127.0.71.124:43719,127.0.71.123:45007 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:58:18.531229 6435 sys_catalog.cc:455] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 9d0a5a9a6599458a88169f472ea39b10. Latest consensus state: current_term: 1 leader_uuid: "9d0a5a9a6599458a88169f472ea39b10" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: false } } }
I20260929 01:58:18.531589 6435 sys_catalog.cc:458] T 00000000000000000000000000000000 P b0a89033ea9c493dbb253bd81a289e1c [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:58:18.840271 6453 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:18.840687 6453 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:18.840814 6453 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:18.850760 6453 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:18.850997 6453 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:58:18.864279 6453 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:39891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=39267
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:33965,127.0.71.125:32867,127.0.71.124:43719,127.0.71.123:45007
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:18.866367 6453 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:18.867985 6453 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:18.880698 6459 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:18.880723 6458 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:18.882040 6461 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:18.882656 6453 server_base.cc:1061] running on GCE node
I20260929 01:58:18.883502 6453 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:58:18.883726 6453 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:58:18.883903 6453 hybrid_clock.cc:648] HybridClock initialized: now 1790647098883878 us; error 0 us; skew 500 ppm
I20260929 01:58:18.884286 6453 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:18 dist-test-slave-vz23 krb5kdc[5575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:18.888813 6453 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:58:18.891093 6453 webserver.cc:533] Webserver started at http://127.0.71.65:39267/ using document root <none> and password file <none>
I20260929 01:58:18.892077 6453 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:18.892243 6453 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:18.899313 6453 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:58:18.904299 6467 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:18.906579 6453 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260929 01:58:18.906873 6453 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "22d7eb8342a54591af883185205b9903"
format_stamp: "Formatted at 2026-09-29 01:58:15 on dist-test-slave-vz23"
I20260929 01:58:18.907873 6453 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:18.933424 6453 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:18.938947 6453 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:18.939322 6453 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:18.940696 6453 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:58:18.942658 6453 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:58:18.942830 6453 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:18.942983 6453 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:58:18.943079 6453 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:18.985663 6453 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:39891
I20260929 01:58:18.985749 6579 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:39891 every 8 connection(s)
I20260929 01:58:18.987748 6453 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:58:18.994843 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 6453
Sep 29 01:58:19 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:58:19 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:58:19 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:58:19 dist-test-slave-vz23 krb5kdc[5575](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647098, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
I20260929 01:58:19.058634 6582 heartbeater.cc:344] Connected to a master server at 127.0.71.126:33965
I20260929 01:58:19.059209 6582 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:19.060866 6582 heartbeater.cc:507] Master 127.0.71.126:33965 requested a full tablet report, sending...
I20260929 01:58:19.063468 6581 heartbeater.cc:344] Connected to a master server at 127.0.71.124:43719
I20260929 01:58:19.063719 6581 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:19.064337 6581 heartbeater.cc:507] Master 127.0.71.124:43719 requested a full tablet report, sending...
I20260929 01:58:19.065500 6394 ts_manager.cc:194] Registered new tserver with Master: 22d7eb8342a54591af883185205b9903 (127.0.71.65:39891)
I20260929 01:58:19.066293 6580 heartbeater.cc:344] Connected to a master server at 127.0.71.123:45007
I20260929 01:58:19.066498 6580 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:19.067199 6580 heartbeater.cc:507] Master 127.0.71.123:45007 requested a full tablet report, sending...
I20260929 01:58:19.067364 5773 ts_manager.cc:194] Re-registered known tserver with Master: 22d7eb8342a54591af883185205b9903 (127.0.71.65:39891)
I20260929 01:58:19.067828 6583 heartbeater.cc:344] Connected to a master server at 127.0.71.125:32867
I20260929 01:58:19.067992 6583 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:19.068657 6583 heartbeater.cc:507] Master 127.0.71.125:32867 requested a full tablet report, sending...
I20260929 01:58:19.069195 6156 ts_manager.cc:194] Re-registered known tserver with Master: 22d7eb8342a54591af883185205b9903 (127.0.71.65:39891)
I20260929 01:58:19.070202 5694 ts_manager.cc:194] Re-registered known tserver with Master: 22d7eb8342a54591af883185205b9903 (127.0.71.65:39891)
I20260929 01:58:19.071120 5694 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:36059
I20260929 01:58:19.096259 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 6453
W20260929 01:58:19.098356 6595 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:58:19.093191 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:58090 (local address 127.0.71.65:39891)
0929 01:58:19.094082 (+ 891us) server_negotiation.cc:207] Beginning negotiation
0929 01:58:19.094091 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0929 01:58:19.094127 (+ 36us) server_negotiation.cc:408] Connection header received
0929 01:58:19.094199 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:58:19.094207 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:58:19.094467 (+ 260us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:58:19.094662 (+ 195us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:58:19.095955 (+ 1293us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58090: BlockingRecv error: recv got EOF from 127.0.0.1:58090 (error 108)
Metrics: {"server-negotiator.queue_time_us":699,"thread_start_us":506,"threads_started":1}
W20260929 01:58:19.100289 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 6453: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:39891: server requires authentication, but client does not have Kerberos credentials available
W20260929 01:58:19.100554 6596 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:58:19.097643 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:58092 (local address 127.0.71.65:39891)
0929 01:58:19.098725 (+ 1082us) server_negotiation.cc:207] Beginning negotiation
0929 01:58:19.098730 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0929 01:58:19.098755 (+ 25us) server_negotiation.cc:408] Connection header received
0929 01:58:19.098821 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:58:19.098826 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:58:19.098933 (+ 107us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:58:19.099099 (+ 166us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:58:19.100201 (+ 1102us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:58092: BlockingRecv error: recv got EOF from 127.0.0.1:58092 (error 108)
Metrics: {"server-negotiator.queue_time_us":914,"thread_start_us":578,"threads_started":1}
I20260929 01:58:19.100610 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 6453
I20260929 01:58:19.117889 6365 sys_catalog.cc:455] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "9d0a5a9a6599458a88169f472ea39b10" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b0a89033ea9c493dbb253bd81a289e1c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33965 } } peers { permanent_uuid: "9d0a5a9a6599458a88169f472ea39b10" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 32867 } } peers { permanent_uuid: "fd42a74e2edd48059be412278430c837" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43719 } } peers { permanent_uuid: "9ece97800ec6402a8930e980360d77a1" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 45007 } attrs { promote: false } } }
I20260929 01:58:19.118337 6365 sys_catalog.cc:458] T 00000000000000000000000000000000 P 9d0a5a9a6599458a88169f472ea39b10 [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:19.122682 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:39891
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:33965,127.0.71.125:32867,127.0.71.124:43719
--webserver_port=39267
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--tserver_master_addrs=127.0.71.126:33965,127.0.71.125:32867,127.0.71.124:43719,127.0.71.123:45007
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:58:19.553330 6600 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:19.553730 6600 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:19.553926 6600 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:58:19.554028 6600 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:19.564070 6600 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:19.564400 6600 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:58:19.577410 6600 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:39891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=39267
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:33965,127.0.71.125:32867,127.0.71.124:43719,127.0.71.123:45007
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:19.579402 6600 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:19.581174 6600 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:19.595039 6605 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:19.594969 6606 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:19.597352 6608 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:19.597677 6600 server_base.cc:1061] running on GCE node
I20260929 01:58:19.598615 6600 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:58:19.598834 6600 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:58:19.598982 6600 hybrid_clock.cc:648] HybridClock initialized: now 1790647099598921 us; error 0 us; skew 500 ppm
I20260929 01:58:19.599378 6600 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:19 dist-test-slave-vz23 krb5kdc[5575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647099, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:19.603791 6600 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:58:19.606139 6600 webserver.cc:533] Webserver started at http://127.0.71.65:39267/ using document root <none> and password file <none>
I20260929 01:58:19.607071 6600 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:19.607178 6600 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:19.614754 6600 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.000s sys 0.004s
I20260929 01:58:19.618832 6614 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:19.620918 6600 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:58:19.621264 6600 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "22d7eb8342a54591af883185205b9903"
format_stamp: "Formatted at 2026-09-29 01:58:15 on dist-test-slave-vz23"
I20260929 01:58:19.622349 6600 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:19.644172 6600 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:19.649255 6600 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:19.649605 6600 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:19.650820 6600 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:58:19.652849 6600 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:58:19.652947 6600 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:19.653090 6600 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:58:19.653146 6600 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:58:19.653664 6600 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:58:19.653759 6600 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:58:19.654000 6600 fault_injection.cc:59] injecting fault for kudu (pid 6600): FLAGS_fault_before_start (process will exit)
I20260929 01:58:19.670989 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:39891
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:33965,127.0.71.125:32867,127.0.71.124:43719
--webserver_port=39267
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--tserver_master_addrs=127.0.71.126:33965,127.0.71.125:32867,127.0.71.124:43719,127.0.71.123:45007 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:58:20.083050 6622 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:20.083673 6622 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:20.083866 6622 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:20.095031 6622 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:20.095408 6622 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:58:20.109555 6622 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:39891
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=39267
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:33965,127.0.71.125:32867,127.0.71.124:43719,127.0.71.123:45007
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:20.111759 6622 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:20.113610 6622 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:20.126986 6628 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:20.127004 6627 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:20.128221 6630 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:20.128947 6622 server_base.cc:1061] running on GCE node
I20260929 01:58:20.129846 6622 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:58:20.130052 6622 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:58:20.130182 6622 hybrid_clock.cc:648] HybridClock initialized: now 1790647100130134 us; error 0 us; skew 500 ppm
I20260929 01:58:20.130533 6622 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:20 dist-test-slave-vz23 krb5kdc[5575](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647100, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:20.134559 6622 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:58:20.137221 6622 webserver.cc:533] Webserver started at http://127.0.71.65:39267/ using document root <none> and password file <none>
I20260929 01:58:20.138175 6622 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:20.138316 6622 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:20.145895 6622 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 01:58:20.150285 6636 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:20.152309 6622 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260929 01:58:20.152654 6622 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "22d7eb8342a54591af883185205b9903"
format_stamp: "Formatted at 2026-09-29 01:58:15 on dist-test-slave-vz23"
I20260929 01:58:20.153620 6622 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:20.180383 6622 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:20.185873 6622 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:20.186200 6622 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:20.187603 6622 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:58:20.189917 6622 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:58:20.190048 6622 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:20.190153 6622 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:58:20.190217 6622 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:20.238174 6622 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:39891
I20260929 01:58:20.238405 6748 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:39891 every 8 connection(s)
I20260929 01:58:20.240248 6622 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:58:20.248615 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 6622
I20260929 01:58:20.248971 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 6622
W20260929 01:58:20.272894 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 6622: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:39891: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:58:20.273094 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 6622
W20260929 01:58:20.293939 6762 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:58:20.263117 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:43419 (local address 127.0.71.126:33965)
0929 01:58:20.269590 (+ 6473us) server_negotiation.cc:207] Beginning negotiation
0929 01:58:20.269600 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0929 01:58:20.292825 (+ 23225us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:43419: BlockingRecv error: recv got EOF from 127.0.71.65:43419 (error 108)
Metrics: {"server-negotiator.queue_time_us":6287,"thread_start_us":6059,"threads_started":1}
W20260929 01:58:20.293943 6756 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:58:20.247263 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:35775 (local address 127.0.71.125:32867)
0929 01:58:20.270595 (+ 23332us) server_negotiation.cc:207] Beginning negotiation
0929 01:58:20.270605 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0929 01:58:20.270637 (+ 32us) server_negotiation.cc:408] Connection header received
0929 01:58:20.270738 (+ 101us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:58:20.270748 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:58:20.270869 (+ 121us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:58:20.271061 (+ 192us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:58:20.292853 (+ 21792us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:35775: BlockingRecv error: recv got EOF from 127.0.71.65:35775 (error 108)
Metrics: {"server-negotiator.queue_time_us":23154,"thread_start_us":23027,"threads_started":1}
W20260929 01:58:20.293927 6760 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:58:20.252154 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:46329 (local address 127.0.71.124:43719)
0929 01:58:20.266952 (+ 14798us) server_negotiation.cc:207] Beginning negotiation
0929 01:58:20.266985 (+ 33us) server_negotiation.cc:400] Waiting for connection header
0929 01:58:20.271183 (+ 4198us) server_negotiation.cc:408] Connection header received
0929 01:58:20.271275 (+ 92us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:58:20.271282 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:58:20.271402 (+ 120us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:58:20.271556 (+ 154us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:58:20.292888 (+ 21332us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:46329: BlockingRecv error: recv got EOF from 127.0.71.65:46329 (error 108)
Metrics: {"server-negotiator.queue_time_us":14591,"thread_start_us":14836,"threads_started":1}
W20260929 01:58:20.293927 6754 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:58:20.244494 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:32789 (local address 127.0.71.123:45007)
0929 01:58:20.275803 (+ 31309us) server_negotiation.cc:207] Beginning negotiation
0929 01:58:20.275811 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0929 01:58:20.275839 (+ 28us) server_negotiation.cc:408] Connection header received
0929 01:58:20.275942 (+ 103us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:58:20.275946 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:58:20.276039 (+ 93us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:58:20.276152 (+ 113us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:58:20.293091 (+ 16939us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:32789: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":31184,"thread_start_us":31047,"threads_started":1}
I20260929 01:58:20.294711 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 6358
W20260929 01:58:20.297610 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 6358: Not authorized: Client connection negotiation failed: client connection to 127.0.71.126:33965: server requires authentication, but client does not have Kerberos credentials available
W20260929 01:58:20.297719 6762 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:58:20.295983 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:60146 (local address 127.0.71.126:33965)
0929 01:58:20.296164 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0929 01:58:20.296169 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0929 01:58:20.296189 (+ 20us) server_negotiation.cc:408] Connection header received
0929 01:58:20.296254 (+ 65us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:58:20.296259 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:58:20.296370 (+ 111us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:58:20.296616 (+ 246us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:58:20.297455 (+ 839us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:60146: BlockingRecv error: recv got EOF from 127.0.0.1:60146 (error 108)
Metrics: {"server-negotiator.queue_time_us":71}
I20260929 01:58:20.297837 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 6358
I20260929 01:58:20.321149 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 5664
I20260929 01:58:20.322724 5725 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:58:20.451499 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 5664
W20260929 01:58:20.477890 5760 connection.cc:573] server connection from 127.0.0.1:38002 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:58:20.478322 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 5740
I20260929 01:58:20.479601 5803 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:58:20.604815 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 5740
I20260929 01:58:20.630505 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 6016
I20260929 01:58:20.632022 6186 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:58:20.758997 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 6016
[ OK ] ExternalMiniClusterTest.TestAddMaster/6 (11445 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/7
Loading random data
Initializing database '/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:58:20 dist-test-slave-vz23 krb5kdc[6769](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:58:20 dist-test-slave-vz23 krb5kdc[6769](info): set up 2 sockets
Sep 29 01:58:20 dist-test-slave-vz23 krb5kdc[6769](info): commencing operation
krb5kdc: starting...
W20260929 01:58:23.001993 285 mini_kdc.cc:121] Time spent starting KDC: real 2.201s user 0.000s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 01:58:23 dist-test-slave-vz23 krb5kdc[6769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:58:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:58:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:58:23.397228 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:41987
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:36809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:41987,127.0.71.125:42953,127.0.71.124:44253 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:58:23.803368 6785 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:23.803776 6785 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:23.803967 6785 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:23.813808 6785 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:58:23.813953 6785 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:23.814023 6785 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:58:23.814069 6785 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:58:23.826570 6785 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:36809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:41987,127.0.71.125:42953,127.0.71.124:44253
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:41987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:23.828536 6785 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:23.830264 6785 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:23.844503 6791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:23.844880 6790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:23.845497 6793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:23.847311 6785 server_base.cc:1061] running on GCE node
I20260929 01:58:23.848984 6785 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:23.852391 6785 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:23.854128 6785 hybrid_clock.cc:648] HybridClock initialized: now 1790647103854025 us; error 105 us; skew 500 ppm
I20260929 01:58:23.854830 6785 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:23 dist-test-slave-vz23 krb5kdc[6769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:23.860121 6785 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:58:23.862996 6785 webserver.cc:533] Webserver started at http://127.0.71.126:42025/ using document root <none> and password file <none>
I20260929 01:58:23.864099 6785 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:23.864306 6785 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:23.864893 6785 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:58:23.868095 6785 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "2c3b21136a584b148e7f6be3709a4077"
format_stamp: "Formatted at 2026-09-29 01:58:23 on dist-test-slave-vz23"
I20260929 01:58:23.869453 6785 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "2c3b21136a584b148e7f6be3709a4077"
format_stamp: "Formatted at 2026-09-29 01:58:23 on dist-test-slave-vz23"
I20260929 01:58:23.878192 6785 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.006s sys 0.001s
I20260929 01:58:23.883854 6800 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:23.886826 6785 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260929 01:58:23.887198 6785 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "2c3b21136a584b148e7f6be3709a4077"
format_stamp: "Formatted at 2026-09-29 01:58:23 on dist-test-slave-vz23"
I20260929 01:58:23.887535 6785 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:23.915802 6785 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:23.927299 6785 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:23.927706 6785 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:23.954207 6785 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:41987
I20260929 01:58:23.954231 6851 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:41987 every 8 connection(s)
I20260929 01:58:23.956179 6785 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:58:23.962177 6852 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:58:23.965725 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 6785
I20260929 01:58:23.965974 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:58:23.971899 6852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:58:23 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.125@KRBTEST.COM" created.
I20260929 01:58:24.004194 6852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:58:24.005772 6804 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.125:42953: connect: Connection refused (error 111)
W20260929 01:58:24.007942 6852 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:42953: Network error: Client connection negotiation failed: client connection to 127.0.71.125:42953: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:58:24.057375 6852 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } attempt: 1
W20260929 01:58:24.060534 6852 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:42953: Network error: Client connection negotiation failed: client connection to 127.0.71.125:42953: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.0.71.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:58:24.121822 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:42953
--webserver_interface=127.0.71.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:36809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:41987,127.0.71.125:42953,127.0.71.124:44253 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:58:24.128943 6852 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } attempt: 2
W20260929 01:58:24.131880 6852 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:42953: Network error: Client connection negotiation failed: client connection to 127.0.71.125:42953: connect: Connection refused (error 111)
I20260929 01:58:24.223263 6852 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } attempt: 3
W20260929 01:58:24.226136 6852 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:42953: Network error: Client connection negotiation failed: client connection to 127.0.71.125:42953: connect: Connection refused (error 111)
I20260929 01:58:24.369546 6852 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } attempt: 4
W20260929 01:58:24.372367 6852 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:42953: Network error: Client connection negotiation failed: client connection to 127.0.71.125:42953: connect: Connection refused (error 111)
W20260929 01:58:24.540490 6861 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:24.540881 6861 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:24.541085 6861 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:24.550568 6861 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:58:24.550720 6861 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:24.550824 6861 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:58:24.550904 6861 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:58:24.563443 6861 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:36809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:41987,127.0.71.125:42953,127.0.71.124:44253
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:42953
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:24.565541 6861 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:24.567159 6861 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:24.577908 6866 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:24.577862 6867 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:24.577883 6869 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:24.578765 6861 server_base.cc:1061] running on GCE node
I20260929 01:58:24.579764 6861 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:24.581763 6861 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:24.583134 6861 hybrid_clock.cc:648] HybridClock initialized: now 1790647104583089 us; error 92 us; skew 500 ppm
I20260929 01:58:24.583698 6861 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:24 dist-test-slave-vz23 krb5kdc[6769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:24.588549 6861 init.cc:377] Logged in from keytab as kudu/127.0.71.125@KRBTEST.COM (short username kudu)
I20260929 01:58:24.590948 6861 webserver.cc:533] Webserver started at http://127.0.71.125:38585/ using document root <none> and password file <none>
I20260929 01:58:24.591918 6861 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:24.592118 6861 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:24.592675 6861 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:58:24.596020 6861 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-1/data/instance:
uuid: "4db60feed0e84478ae90c5b868d0e44d"
format_stamp: "Formatted at 2026-09-29 01:58:24 on dist-test-slave-vz23"
I20260929 01:58:24.597078 6861 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-1/wal/instance:
uuid: "4db60feed0e84478ae90c5b868d0e44d"
format_stamp: "Formatted at 2026-09-29 01:58:24 on dist-test-slave-vz23"
I20260929 01:58:24.604415 6861 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.000s
I20260929 01:58:24.609731 6876 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:24.611706 6861 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:58:24.612016 6861 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "4db60feed0e84478ae90c5b868d0e44d"
format_stamp: "Formatted at 2026-09-29 01:58:24 on dist-test-slave-vz23"
I20260929 01:58:24.612277 6861 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:24.634250 6861 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:24.639744 6861 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:24.640097 6861 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:24.664778 6861 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:42953
I20260929 01:58:24.664829 6927 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:42953 every 8 connection(s)
I20260929 01:58:24.666822 6861 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:58:24.667907 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 6861
I20260929 01:58:24.668488 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:58:24.671795 6852 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } attempt: 5
I20260929 01:58:24.674549 6928 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:58:24.685286 6928 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:58:24 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.124@KRBTEST.COM; defaulting to no policy
I20260929 01:58:24.708820 6852 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:58:24.711727 6852 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:44253: Network error: Client connection negotiation failed: client connection to 127.0.71.124:44253: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.124@KRBTEST.COM" created.
Sep 29 01:58:24 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:58:24.725370 6928 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:58:24 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:58:24.741017 6928 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:58:24.742630 6880 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.124:44253: connect: Connection refused (error 111)
W20260929 01:58:24.745507 6928 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:44253: Network error: Client connection negotiation failed: client connection to 127.0.71.124:44253: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:58:24.763125 6852 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } attempt: 1
W20260929 01:58:24.765836 6852 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:44253: Network error: Client connection negotiation failed: client connection to 127.0.71.124:44253: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.0.71.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.124@KRBTEST.COM" created.
I20260929 01:58:24.795161 6928 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } attempt: 1
W20260929 01:58:24.798022 6928 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:44253: Network error: Client connection negotiation failed: client connection to 127.0.71.124:44253: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:58:24.831043 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:44253
--webserver_interface=127.0.71.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:36809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:41987,127.0.71.125:42953,127.0.71.124:44253 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:58:24.834235 6852 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } attempt: 2
W20260929 01:58:24.837041 6852 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:44253: Network error: Client connection negotiation failed: client connection to 127.0.71.124:44253: connect: Connection refused (error 111)
I20260929 01:58:24.866447 6928 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } attempt: 2
W20260929 01:58:24.869377 6928 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:44253: Network error: Client connection negotiation failed: client connection to 127.0.71.124:44253: connect: Connection refused (error 111)
I20260929 01:58:24.943460 6852 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } attempt: 3
W20260929 01:58:24.946442 6852 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:44253: Network error: Client connection negotiation failed: client connection to 127.0.71.124:44253: connect: Connection refused (error 111)
I20260929 01:58:24.960668 6928 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } attempt: 3
W20260929 01:58:24.963546 6928 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:44253: Network error: Client connection negotiation failed: client connection to 127.0.71.124:44253: connect: Connection refused (error 111)
I20260929 01:58:25.106860 6928 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } attempt: 4
W20260929 01:58:25.109913 6928 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:44253: Network error: Client connection negotiation failed: client connection to 127.0.71.124:44253: connect: Connection refused (error 111)
I20260929 01:58:25.123749 6852 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } attempt: 4
W20260929 01:58:25.126644 6852 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:44253: Network error: Client connection negotiation failed: client connection to 127.0.71.124:44253: connect: Connection refused (error 111)
W20260929 01:58:25.253654 6938 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:25.254124 6938 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:25.254298 6938 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:25.264521 6938 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:58:25.264734 6938 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:25.264855 6938 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:58:25.264945 6938 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:58:25.277614 6938 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:36809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:41987,127.0.71.125:42953,127.0.71.124:44253
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:44253
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:25.279706 6938 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:25.281502 6938 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:25.291811 6943 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:25.291674 6944 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:25.293551 6946 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:25.294328 6938 server_base.cc:1061] running on GCE node
I20260929 01:58:25.295404 6938 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:25.297657 6938 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:25.299013 6938 hybrid_clock.cc:648] HybridClock initialized: now 1790647105298943 us; error 73 us; skew 500 ppm
I20260929 01:58:25.299612 6938 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:25 dist-test-slave-vz23 krb5kdc[6769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:25.304234 6938 init.cc:377] Logged in from keytab as kudu/127.0.71.124@KRBTEST.COM (short username kudu)
I20260929 01:58:25.306656 6938 webserver.cc:533] Webserver started at http://127.0.71.124:46761/ using document root <none> and password file <none>
I20260929 01:58:25.307615 6938 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:25.307799 6938 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:25.308236 6938 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:58:25.311314 6938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-2/data/instance:
uuid: "789c69f6f65148bb96baf03e70e86fc4"
format_stamp: "Formatted at 2026-09-29 01:58:25 on dist-test-slave-vz23"
I20260929 01:58:25.312332 6938 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-2/wal/instance:
uuid: "789c69f6f65148bb96baf03e70e86fc4"
format_stamp: "Formatted at 2026-09-29 01:58:25 on dist-test-slave-vz23"
I20260929 01:58:25.319383 6938 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260929 01:58:25.324568 6953 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:25.326979 6938 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260929 01:58:25.327322 6938 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "789c69f6f65148bb96baf03e70e86fc4"
format_stamp: "Formatted at 2026-09-29 01:58:25 on dist-test-slave-vz23"
I20260929 01:58:25.327585 6938 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:25.342773 6938 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:25.347914 6938 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:25.348305 6938 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:25.373986 6938 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:44253
I20260929 01:58:25.374086 7004 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:44253 every 8 connection(s)
I20260929 01:58:25.376305 6938 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:58:25.378949 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 6938
I20260929 01:58:25.379186 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:58:25.382875 7005 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 01:58:25 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:58:25.397118 7005 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:58:25.403968 6852 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } attempt: 5
I20260929 01:58:25.409194 6928 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } attempt: 5
Sep 29 01:58:25 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:58:25 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647103, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:58:25 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:58:25 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:58:25.431542 7005 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:58:25.441793 6852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077: Bootstrap starting.
Sep 29 01:58:25 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:58:25.447639 6928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d: Bootstrap starting.
I20260929 01:58:25.447701 6852 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077: Neither blocks nor log segments found. Creating new log.
I20260929 01:58:25.449622 6852 log.cc:826] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077: Log is configured to *not* fsync() on all Append() calls
I20260929 01:58:25.453055 6928 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d: Neither blocks nor log segments found. Creating new log.
I20260929 01:58:25.455339 6928 log.cc:826] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:58:25.455413 7005 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:58:25.461869 6852 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077: No bootstrap required, opened a new log
I20260929 01:58:25.464331 6928 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d: No bootstrap required, opened a new log
Sep 29 01:58:25 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647105, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:58:25.471537 6852 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } }
I20260929 01:58:25.472034 6852 raft_consensus.cc:385] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:58:25.472224 6852 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c3b21136a584b148e7f6be3709a4077, State: Initialized, Role: FOLLOWER
I20260929 01:58:25.473331 6852 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } }
I20260929 01:58:25.474272 6928 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } }
I20260929 01:58:25.474718 6928 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:58:25.474871 6928 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4db60feed0e84478ae90c5b868d0e44d, State: Initialized, Role: FOLLOWER
I20260929 01:58:25.476024 6928 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } }
I20260929 01:58:25.483016 6852 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 01:58:25 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:58:25.484048 7019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } }
I20260929 01:58:25.484422 7019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:25.484876 7018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } }
I20260929 01:58:25.485216 7018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:25.486735 6928 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:58:25.488839 7005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4: Bootstrap starting.
I20260929 01:58:25.497084 7005 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4: Neither blocks nor log segments found. Creating new log.
I20260929 01:58:25.501272 7005 log.cc:826] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4: Log is configured to *not* fsync() on all Append() calls
I20260929 01:58:25.506882 7005 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4: No bootstrap required, opened a new log
I20260929 01:58:25.520926 7005 raft_consensus.cc:359] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } }
I20260929 01:58:25.521699 7005 raft_consensus.cc:385] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:58:25.521956 7005 raft_consensus.cc:740] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 789c69f6f65148bb96baf03e70e86fc4, State: Initialized, Role: FOLLOWER
I20260929 01:58:25.523188 7005 consensus_queue.cc:260] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } }
I20260929 01:58:25.530704 7038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } }
I20260929 01:58:25.533725 7038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:25.533507 7005 sys_catalog.cc:565] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:58:25.525188 6861 master_runner.cc:187] Error getting master registration for 127.0.71.126:41987: OK, instance_id { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" instance_seqno: 1790647103947065 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:58:25.542585 6861 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:58:25.554486 7036 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:58:25.554752 7036 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:58:25.572369 6785 master_runner.cc:187] Error getting master registration for 127.0.71.124:44253: OK, instance_id { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" instance_seqno: 1790647105366968 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:58:25.580260 6785 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:58:25.586814 7019 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:58:25.587284 7019 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } }
W20260929 01:58:25.588972 7049 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:58:25.589164 7049 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:58:25.590286 7019 leader_election.cc:290] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 2c3b21136a584b148e7f6be3709a4077 (127.0.71.126:41987), 789c69f6f65148bb96baf03e70e86fc4 (127.0.71.124:44253)
I20260929 01:58:25.592293 6827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4db60feed0e84478ae90c5b868d0e44d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c3b21136a584b148e7f6be3709a4077" is_pre_election: true
I20260929 01:58:25.593276 6827 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4db60feed0e84478ae90c5b868d0e44d in term 0.
I20260929 01:58:25.595144 6880 leader_election.cc:304] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c3b21136a584b148e7f6be3709a4077, 4db60feed0e84478ae90c5b868d0e44d; no voters:
I20260929 01:58:25.597352 6980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4db60feed0e84478ae90c5b868d0e44d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789c69f6f65148bb96baf03e70e86fc4" is_pre_election: true
I20260929 01:58:25.598234 6980 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 4db60feed0e84478ae90c5b868d0e44d in term 0.
I20260929 01:58:25.599962 7019 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:58:25.600286 7019 raft_consensus.cc:493] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:58:25.600502 7019 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [term 0 FOLLOWER]: Advancing to term 1
W20260929 01:58:25.600878 6959 tablet.cc:2433] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:58:25.605683 7019 raft_consensus.cc:515] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } }
I20260929 01:58:25.611649 6827 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4db60feed0e84478ae90c5b868d0e44d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "2c3b21136a584b148e7f6be3709a4077"
I20260929 01:58:25.612061 6827 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:58:25.618294 6827 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4db60feed0e84478ae90c5b868d0e44d in term 1.
I20260929 01:58:25.619356 6880 leader_election.cc:304] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 2c3b21136a584b148e7f6be3709a4077, 4db60feed0e84478ae90c5b868d0e44d; no voters:
I20260929 01:58:25.630759 7019 leader_election.cc:290] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [CANDIDATE]: Term 1 election: Requested vote from peers 2c3b21136a584b148e7f6be3709a4077 (127.0.71.126:41987), 789c69f6f65148bb96baf03e70e86fc4 (127.0.71.124:44253)
I20260929 01:58:25.631223 7019 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:58:25.631454 7019 raft_consensus.cc:697] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [term 1 LEADER]: Becoming Leader. State: Replica: 4db60feed0e84478ae90c5b868d0e44d, State: Running, Role: LEADER
I20260929 01:58:25.632258 7019 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } }
I20260929 01:58:25.641812 6980 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "4db60feed0e84478ae90c5b868d0e44d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "789c69f6f65148bb96baf03e70e86fc4"
I20260929 01:58:25.642160 6980 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4 [term 0 FOLLOWER]: Advancing to term 1
W20260929 01:58:25.644809 6882 tablet.cc:2433] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:58:25.655184 6980 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 4db60feed0e84478ae90c5b868d0e44d in term 1.
I20260929 01:58:25.659940 7057 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4db60feed0e84478ae90c5b868d0e44d. Latest consensus state: current_term: 1 leader_uuid: "4db60feed0e84478ae90c5b868d0e44d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } }
I20260929 01:58:25.661177 7057 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:25.663285 7059 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:58:25.667023 7059 catalog_manager.cc:1533] Initializing Kudu cluster ID...
W20260929 01:58:25.680655 7056 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:58:25.680868 7056 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 01:58:25.683089 6806 tablet.cc:2433] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:58:25.702728 6980 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4 [term 1 FOLLOWER]: Refusing update from remote peer 4db60feed0e84478ae90c5b868d0e44d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:58:25.704180 7057 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [LEADER]: Connected to new peer: Peer: permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:58:25.716645 6827 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [term 1 FOLLOWER]: Refusing update from remote peer 4db60feed0e84478ae90c5b868d0e44d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:58:25.718170 7057 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:58:25.755900 7038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4db60feed0e84478ae90c5b868d0e44d. Latest consensus state: current_term: 1 leader_uuid: "4db60feed0e84478ae90c5b868d0e44d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } }
I20260929 01:58:25.756253 7038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:25.764104 7062 mvcc.cc:204] Tried to move back new op lower bound from 7334490544944345088 to 7334490544689250304. Current Snapshot: MvccSnapshot[applied={T|T < 7334490544944345088}]
I20260929 01:58:25.765820 7018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4db60feed0e84478ae90c5b868d0e44d. Latest consensus state: current_term: 1 leader_uuid: "4db60feed0e84478ae90c5b868d0e44d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } }
I20260929 01:58:25.766199 7018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:25.784512 7019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4db60feed0e84478ae90c5b868d0e44d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } }
I20260929 01:58:25.784969 7019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:25.784966 7018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4db60feed0e84478ae90c5b868d0e44d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } }
I20260929 01:58:25.785244 7018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:25.785317 7019 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4db60feed0e84478ae90c5b868d0e44d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } }
I20260929 01:58:25.785599 7019 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:25.788108 7059 catalog_manager.cc:1396] Generated new cluster ID: e10a6bfc0e2a4b2ba755de70bc579c20
I20260929 01:58:25.788331 7059 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:58:25.789590 7038 sys_catalog.cc:455] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4db60feed0e84478ae90c5b868d0e44d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } }
I20260929 01:58:25.789839 7038 sys_catalog.cc:458] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:25.826426 7059 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 01:58:25.828631 7059 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:58:25.866199 7059 catalog_manager.cc:6219] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d: Generated new TSK 0
I20260929 01:58:25.867785 7059 catalog_manager.cc:1563] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
W20260929 01:58:26.024077 285 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.0.71.65: real 0.133s user 0.001s sys 0.027s
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
W20260929 01:58:26.144692 285 mini_kdc.cc:268] Time spent creating service keytab for HTTP/127.0.71.65: real 0.120s user 0.007s sys 0.021s
I20260929 01:58:26.145987 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:41987,127.0.71.125:42953,127.0.71.124:44253
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:36809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:58:26.592059 7049 catalog_manager.cc:1308] Loaded cluster ID: e10a6bfc0e2a4b2ba755de70bc579c20
I20260929 01:58:26.592228 7049 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077: loading cluster ID for follower catalog manager: success
I20260929 01:58:26.597043 7049 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077: acquiring CA information for follower catalog manager: success
I20260929 01:58:26.600641 7049 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 01:58:26.651643 7073 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:26.652022 7073 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:26.652127 7073 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:26.661721 7073 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:26.661963 7073 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:58:26.680166 7073 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:36809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:41987,127.0.71.125:42953,127.0.71.124:44253
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:26.682771 7073 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:26.683146 7056 catalog_manager.cc:1308] Loaded cluster ID: e10a6bfc0e2a4b2ba755de70bc579c20
I20260929 01:58:26.683271 7056 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4: loading cluster ID for follower catalog manager: success
I20260929 01:58:26.684934 7073 file_cache.cc:504] Constructed file cache file cache with capacity 419430
I20260929 01:58:26.686744 7056 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4: acquiring CA information for follower catalog manager: success
I20260929 01:58:26.689244 7056 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 01:58:26.698786 7079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:26.699081 7081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:26.699463 7078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:26.699606 7073 server_base.cc:1061] running on GCE node
I20260929 01:58:26.700378 7073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:26.701853 7073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:26.703200 7073 hybrid_clock.cc:648] HybridClock initialized: now 1790647106703144 us; error 89 us; skew 500 ppm
I20260929 01:58:26.703711 7073 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:26 dist-test-slave-vz23 krb5kdc[6769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:26.708777 7073 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:58:26.711129 7073 webserver.cc:533] Webserver started at http://127.0.71.65:35241/ using document root <none> and password file <none>
I20260929 01:58:26.712145 7073 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:26.712348 7073 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:26.712868 7073 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:58:26.716058 7073 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "faf5e50f688b449b825d4a398a6508d9"
format_stamp: "Formatted at 2026-09-29 01:58:26 on dist-test-slave-vz23"
I20260929 01:58:26.717293 7073 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "faf5e50f688b449b825d4a398a6508d9"
format_stamp: "Formatted at 2026-09-29 01:58:26 on dist-test-slave-vz23"
I20260929 01:58:26.725379 7073 fs_manager.cc:696] Time spent creating directory manager: real 0.008s user 0.007s sys 0.000s
I20260929 01:58:26.730499 7088 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:26.732429 7073 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:58:26.732761 7073 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "faf5e50f688b449b825d4a398a6508d9"
format_stamp: "Formatted at 2026-09-29 01:58:26 on dist-test-slave-vz23"
I20260929 01:58:26.733034 7073 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:26.758306 7073 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:26.764104 7073 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:26.764658 7073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:26.766088 7073 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:58:26.768379 7073 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:58:26.768569 7073 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.001s
I20260929 01:58:26.768703 7073 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:58:26.768764 7073 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:26.812350 7073 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:35227
I20260929 01:58:26.812412 7200 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:35227 every 8 connection(s)
I20260929 01:58:26.814342 7073 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:58:26.816421 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 7073
I20260929 01:58:26.816697 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
Sep 29 01:58:26 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:58:26 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:58:26 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647106, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:58:26.856387 7202 heartbeater.cc:344] Connected to a master server at 127.0.71.126:41987
I20260929 01:58:26.856331 7201 heartbeater.cc:344] Connected to a master server at 127.0.71.124:44253
I20260929 01:58:26.856422 7203 heartbeater.cc:344] Connected to a master server at 127.0.71.125:42953
I20260929 01:58:26.857148 7202 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:26.857148 7203 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:26.857146 7201 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:26.858775 7202 heartbeater.cc:507] Master 127.0.71.126:41987 requested a full tablet report, sending...
I20260929 01:58:26.858840 7201 heartbeater.cc:507] Master 127.0.71.124:44253 requested a full tablet report, sending...
I20260929 01:58:26.858872 7203 heartbeater.cc:507] Master 127.0.71.125:42953 requested a full tablet report, sending...
I20260929 01:58:26.862735 6969 ts_manager.cc:194] Registered new tserver with Master: faf5e50f688b449b825d4a398a6508d9 (127.0.71.65:35227)
I20260929 01:58:26.862718 6893 ts_manager.cc:194] Registered new tserver with Master: faf5e50f688b449b825d4a398a6508d9 (127.0.71.65:35227)
I20260929 01:58:26.864912 6817 ts_manager.cc:194] Registered new tserver with Master: faf5e50f688b449b825d4a398a6508d9 (127.0.71.65:35227)
I20260929 01:58:26.865259 6893 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:39963
I20260929 01:58:26.877621 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
WARNING: no policy specified for kudu/127.0.71.123@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.123@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.123 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.123 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.123@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.123@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.123 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.123 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:58:27.034144 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.123:44065
--webserver_interface=127.0.71.123
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.123
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:36809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:41987,127.0.71.125:42953,127.0.71.124:44253,127.0.71.123:44065 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:58:27.461885 7216 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:27.462407 7216 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:27.462574 7216 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:27.472255 7216 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:58:27.472520 7216 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:27.472656 7216 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:58:27.472752 7216 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:58:27.486238 7216 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:36809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:41987,127.0.71.125:42953,127.0.71.124:44253,127.0.71.123:44065
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.123
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.123:44065
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.0.71.123
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:27.489509 7216 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:27.491271 7216 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:27.503019 7221 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:27.503072 7222 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:27.505028 7224 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:27.504938 7216 server_base.cc:1061] running on GCE node
I20260929 01:58:27.506377 7216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:27.508865 7216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:27.510277 7216 hybrid_clock.cc:648] HybridClock initialized: now 1790647107510209 us; error 57 us; skew 500 ppm
I20260929 01:58:27.510839 7216 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:27 dist-test-slave-vz23 krb5kdc[6769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:27.515466 7216 init.cc:377] Logged in from keytab as kudu/127.0.71.123@KRBTEST.COM (short username kudu)
I20260929 01:58:27.517841 7216 webserver.cc:533] Webserver started at http://127.0.71.123:43883/ using document root <none> and password file <none>
I20260929 01:58:27.518890 7216 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:27.519029 7216 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:27.519481 7216 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:58:27.522722 7216 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-3/data/instance:
uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e"
format_stamp: "Formatted at 2026-09-29 01:58:27 on dist-test-slave-vz23"
I20260929 01:58:27.523730 7216 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-3/wal/instance:
uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e"
format_stamp: "Formatted at 2026-09-29 01:58:27 on dist-test-slave-vz23"
I20260929 01:58:27.531303 7216 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.004s
I20260929 01:58:27.536820 7231 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:27.538928 7216 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260929 01:58:27.539252 7216 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e"
format_stamp: "Formatted at 2026-09-29 01:58:27 on dist-test-slave-vz23"
I20260929 01:58:27.539602 7216 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:27.564702 7216 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:27.570380 7216 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:27.570808 7216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:27.597878 7216 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.123:44065
I20260929 01:58:27.597899 7282 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.123:44065 every 8 connection(s)
I20260929 01:58:27.599996 7216 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
I20260929 01:58:27.601645 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 7216
I20260929 01:58:27.607034 7283 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:58:27.614563 7283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:58:27 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
Sep 29 01:58:27 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:58:27.637789 7283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:58:27 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:58:27.654922 7283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:58:27 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:58:27.673529 7283 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44065 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:58:27 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647107, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
I20260929 01:58:27.694839 7283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e: Bootstrap starting.
I20260929 01:58:27.699172 7283 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e: Neither blocks nor log segments found. Creating new log.
I20260929 01:58:27.700824 7283 log.cc:826] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e: Log is configured to *not* fsync() on all Append() calls
I20260929 01:58:27.704218 7283 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e: No bootstrap required, opened a new log
I20260929 01:58:27.710160 7283 raft_consensus.cc:359] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44065 } }
I20260929 01:58:27.710561 7283 raft_consensus.cc:385] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:58:27.710716 7283 raft_consensus.cc:740] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 48ea3ceb72a14cda8d9b5a56c7bbfd3e, State: Initialized, Role: FOLLOWER
I20260929 01:58:27.711580 7283 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44065 } }
I20260929 01:58:27.712853 7292 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44065 } } }
I20260929 01:58:27.713268 7292 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:27.714063 7283 sys_catalog.cc:565] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:58:27.725731 7303 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:58:27.725955 7303 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:58:27.737692 7216 master_runner.cc:429] Detected that this master 48ea3ceb72a14cda8d9b5a56c7bbfd3e is joining an existing cluster
I20260929 01:58:27.737902 7216 catalog_manager.cc:1037] Shutting down the Catalog Manager
I20260929 01:58:27.738417 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 7073
I20260929 01:58:27.739338 7216 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e [term 0 FOLLOWER]: Raft consensus shutting down.
I20260929 01:58:27.739949 7216 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e [term 0 FOLLOWER]: Raft consensus is shut down!
I20260929 01:58:27.740195 7216 tablet_replica.cc:333] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e: stopping tablet replica
I20260929 01:58:27.743144 7216 master_runner.cc:434] Initiating AddMaster RPC to add 127.0.71.123:44065
Sep 29 01:58:27 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647103, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 01:58:27.774474 7196 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 29 01:58:27 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647104, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
I20260929 01:58:27.853397 6893 catalog_manager.cc:7351] Initiating ChangeConfig request to add master 127.0.71.123:44065: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e"
member_type: NON_VOTER
last_known_addr {
host: "127.0.71.123"
port: 44065
}
attrs {
promote: true
}
}
dest_uuid: "4db60feed0e84478ae90c5b868d0e44d"
cas_config_opid_index: -1
I20260929 01:58:27.855440 6893 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: true } }
W20260929 01:58:27.860707 6880 consensus_peers.cc:597] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d -> Peer 48ea3ceb72a14cda8d9b5a56c7bbfd3e (127.0.71.123:44065): Couldn't send request to peer 48ea3ceb72a14cda8d9b5a56c7bbfd3e. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260929 01:58:27.862383 6827 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [term 1 FOLLOWER]: Refusing update from remote peer 4db60feed0e84478ae90c5b868d0e44d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 01:58:27.863245 6979 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4 [term 1 FOLLOWER]: Refusing update from remote peer 4db60feed0e84478ae90c5b868d0e44d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 01:58:27.863343 7213 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:58:27.864068 7213 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [LEADER]: Connected to new peer: Peer: permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:58:27.869065 7317 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 48ea3ceb72a14cda8d9b5a56c7bbfd3e (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: true } } }
I20260929 01:58:27.870208 6827 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 48ea3ceb72a14cda8d9b5a56c7bbfd3e (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: true } } }
I20260929 01:58:27.874644 7317 catalog_manager.cc:7355] Successfully completed master ChangeConfig request to add master 127.0.71.123:44065
I20260929 01:58:27.875700 7216 master.cc:562] Master@127.0.71.123:44065 shutting down...
I20260929 01:58:27.881515 7317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4db60feed0e84478ae90c5b868d0e44d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: true } } }
I20260929 01:58:27.881863 7317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:27.882850 7213 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4db60feed0e84478ae90c5b868d0e44d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: true } } }
I20260929 01:58:27.883255 7213 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:27.906502 7320 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4db60feed0e84478ae90c5b868d0e44d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: true } } }
I20260929 01:58:27.906978 7320 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:27.908501 6979 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 48ea3ceb72a14cda8d9b5a56c7bbfd3e (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: true } } }
I20260929 01:58:27.915810 7324 sys_catalog.cc:455] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4db60feed0e84478ae90c5b868d0e44d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: true } } }
I20260929 01:58:27.916252 7324 sys_catalog.cc:458] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:27.917234 7323 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4db60feed0e84478ae90c5b868d0e44d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: true } } }
I20260929 01:58:27.917651 7323 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:27.938678 7216 master.cc:584] Master@127.0.71.123:44065 shutdown complete.
I20260929 01:58:27.938844 7216 master_runner.cc:306] Clearing existing system tablet
I20260929 01:58:27.939337 7216 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:27.939500 7216 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:27.942390 7216 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.004s sys 0.000s
I20260929 01:58:27.944525 7326 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:27.945041 7216 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:27.945374 7216 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260929 01:58:27.945516 7216 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e"
format_stamp: "Formatted at 2026-09-29 01:58:27 on dist-test-slave-vz23"
I20260929 01:58:27.949404 7216 ts_tablet_manager.cc:1962] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:58:27.956321 7216 ts_tablet_manager.cc:1975] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 01:58:27.956658 7216 log.cc:1199] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260929 01:58:27.957190 7216 ts_tablet_manager.cc:1996] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e: Deleting consensus metadata
I20260929 01:58:27.957551 7216 master_runner.cc:316] Copying system tablet from 127.0.71.125:42953
I20260929 01:58:27.981719 7203 heartbeater.cc:499] Master 127.0.71.125:42953 was elected leader, sending a full tablet report...
I20260929 01:58:27.984627 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 7073
I20260929 01:58:27.989033 7216 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e: tablet copy: Beginning tablet copy session from remote peer at address 127.0.71.125:42953
W20260929 01:58:28.009534 6956 connection.cc:573] server connection from 127.0.71.65:52367 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:58:28.011296 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:35227
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:41987,127.0.71.125:42953,127.0.71.124:44253
--webserver_port=35241
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:36809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:41987,127.0.71.125:42953,127.0.71.124:44253,127.0.71.123:44065 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:58:28.018010 6913 tablet_copy_service.cc:140] P 4db60feed0e84478ae90c5b868d0e44d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 48ea3ceb72a14cda8d9b5a56c7bbfd3e ({username='kudu', principal='kudu/127.0.71.123@KRBTEST.COM'} at 127.0.0.1:47870)
I20260929 01:58:28.018477 6913 tablet_copy_service.cc:161] P 4db60feed0e84478ae90c5b868d0e44d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 48ea3ceb72a14cda8d9b5a56c7bbfd3e at {username='kudu', principal='kudu/127.0.71.123@KRBTEST.COM'} at 127.0.0.1:47870: session id = 48ea3ceb72a14cda8d9b5a56c7bbfd3e-00000000000000000000000000000000
I20260929 01:58:28.023782 6913 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 01:58:28.027498 7216 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 01:58:28.037979 7216 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e: tablet copy: Starting download of 0 data blocks...
I20260929 01:58:28.038455 7216 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e: tablet copy: Starting download of 1 WAL segments...
I20260929 01:58:28.042655 7216 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 01:58:28.049907 6913 tablet_copy_service.cc:342] P 4db60feed0e84478ae90c5b868d0e44d: Request end of tablet copy session 48ea3ceb72a14cda8d9b5a56c7bbfd3e-00000000000000000000000000000000 received from {username='kudu', principal='kudu/127.0.71.123@KRBTEST.COM'} at 127.0.0.1:47870
I20260929 01:58:28.050268 6913 tablet_copy_service.cc:434] P 4db60feed0e84478ae90c5b868d0e44d: ending tablet copy session 48ea3ceb72a14cda8d9b5a56c7bbfd3e-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 48ea3ceb72a14cda8d9b5a56c7bbfd3e
W20260929 01:58:28.063549 7216 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260929 01:58:28.069427 7216 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:28.074025 7334 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:28.074748 7335 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:28.081588 7337 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:28.086412 7216 server_base.cc:1061] running on GCE node
I20260929 01:58:28.086880 7216 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:28.087904 7216 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:28.089294 7216 hybrid_clock.cc:648] HybridClock initialized: now 1790647108089225 us; error 59 us; skew 500 ppm
I20260929 01:58:28.089637 7216 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:28 dist-test-slave-vz23 krb5kdc[6769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:28.092638 7216 init.cc:377] Logged in from keytab as kudu/127.0.71.123@KRBTEST.COM (short username kudu)
I20260929 01:58:28.098045 7216 webserver.cc:533] Webserver started at http://127.0.71.123:40257/ using document root <none> and password file <none>
I20260929 01:58:28.098578 7216 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:28.098806 7216 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:28.102753 7216 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:58:28.105527 7344 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:28.106393 7216 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.001s
I20260929 01:58:28.106583 7216 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e"
format_stamp: "Formatted at 2026-09-29 01:58:27 on dist-test-slave-vz23"
I20260929 01:58:28.106793 7216 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:28.127566 7216 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:28.132912 7216 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:28.165683 7216 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.123:44065
I20260929 01:58:28.165772 7395 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.123:44065 every 8 connection(s)
I20260929 01:58:28.167032 7216 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
I20260929 01:58:28.173125 7396 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:58:28.177999 7396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e: Bootstrap starting.
I20260929 01:58:28.201112 7396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:58:28.201790 7396 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e: Bootstrap complete.
I20260929 01:58:28.203142 7396 raft_consensus.cc:359] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: true } }
I20260929 01:58:28.203444 7396 raft_consensus.cc:740] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 48ea3ceb72a14cda8d9b5a56c7bbfd3e, State: Initialized, Role: LEARNER
I20260929 01:58:28.203784 7396 consensus_queue.cc:260] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: true } }
I20260929 01:58:28.204968 7399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: true } } }
I20260929 01:58:28.205410 7396 sys_catalog.cc:565] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:58:28.205598 7399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e [sys.catalog]: This master's current role is: LEARNER
I20260929 01:58:28.217809 7410 catalog_manager.cc:1308] Loaded cluster ID: e10a6bfc0e2a4b2ba755de70bc579c20
I20260929 01:58:28.217963 7410 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e: loading cluster ID for follower catalog manager: success
I20260929 01:58:28.222940 7410 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e: acquiring CA information for follower catalog manager: success
I20260929 01:58:28.226527 7410 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:58:28.355834 7371 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260929 01:58:28.356949 7399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4db60feed0e84478ae90c5b868d0e44d. Latest consensus state: current_term: 1 leader_uuid: "4db60feed0e84478ae90c5b868d0e44d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: true } } }
I20260929 01:58:28.357280 7399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e [sys.catalog]: This master's current role is: LEARNER
W20260929 01:58:28.510735 7333 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:28.511248 7333 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:28.511431 7333 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:28.522464 7333 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:28.522764 7333 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:58:28.535990 7333 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:36809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:35227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=35241
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:41987,127.0.71.125:42953,127.0.71.124:44253,127.0.71.123:44065
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:28.537976 7333 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:28.539547 7333 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:28.552294 7416 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:28.552544 7417 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:28.552277 7419 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:28.553553 7333 server_base.cc:1061] running on GCE node
I20260929 01:58:28.554328 7333 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:28.555668 7333 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:28.557068 7333 hybrid_clock.cc:648] HybridClock initialized: now 1790647108557006 us; error 68 us; skew 500 ppm
I20260929 01:58:28.557611 7333 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:28 dist-test-slave-vz23 krb5kdc[6769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:28.562026 7333 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:58:28.564559 7333 webserver.cc:533] Webserver started at http://127.0.71.65:35241/ using document root <none> and password file <none>
I20260929 01:58:28.565575 7333 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:28.565809 7333 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:28.573012 7333 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.002s
I20260929 01:58:28.577001 7426 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:28.579051 7333 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:58:28.579340 7333 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "faf5e50f688b449b825d4a398a6508d9"
format_stamp: "Formatted at 2026-09-29 01:58:26 on dist-test-slave-vz23"
I20260929 01:58:28.580271 7333 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:28.603873 7333 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:28.609726 7333 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:28.610090 7333 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:28.611410 7333 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:58:28.613435 7333 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:58:28.613592 7333 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:28.613744 7333 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:58:28.613845 7333 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:28.656901 7333 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:35227
I20260929 01:58:28.656960 7538 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:35227 every 8 connection(s)
I20260929 01:58:28.658970 7333 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:58:28.668361 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 7333
Sep 29 01:58:28 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
Sep 29 01:58:28 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:58:28.717005 7539 heartbeater.cc:344] Connected to a master server at 127.0.71.123:44065
I20260929 01:58:28.717558 7539 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:28.719162 7539 heartbeater.cc:507] Master 127.0.71.123:44065 requested a full tablet report, sending...
Sep 29 01:58:28 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:58:28.724236 7360 ts_manager.cc:194] Registered new tserver with Master: faf5e50f688b449b825d4a398a6508d9 (127.0.71.65:35227)
I20260929 01:58:28.724679 7540 heartbeater.cc:344] Connected to a master server at 127.0.71.124:44253
I20260929 01:58:28.724874 7540 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:28.725525 7540 heartbeater.cc:507] Master 127.0.71.124:44253 requested a full tablet report, sending...
Sep 29 01:58:28 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647108, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:58:28.727933 6969 ts_manager.cc:194] Re-registered known tserver with Master: faf5e50f688b449b825d4a398a6508d9 (127.0.71.65:35227)
I20260929 01:58:28.730577 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:58:28.733028 7542 heartbeater.cc:344] Connected to a master server at 127.0.71.125:42953
I20260929 01:58:28.733256 7542 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:28.734002 7542 heartbeater.cc:507] Master 127.0.71.125:42953 requested a full tablet report, sending...
I20260929 01:58:28.735788 6893 ts_manager.cc:194] Re-registered known tserver with Master: faf5e50f688b449b825d4a398a6508d9 (127.0.71.65:35227)
I20260929 01:58:28.737849 6893 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:49527
I20260929 01:58:28.738245 7541 heartbeater.cc:344] Connected to a master server at 127.0.71.126:41987
I20260929 01:58:28.738456 7541 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:28.739333 7541 heartbeater.cc:507] Master 127.0.71.126:41987 requested a full tablet report, sending...
I20260929 01:58:28.741364 6817 ts_manager.cc:194] Re-registered known tserver with Master: faf5e50f688b449b825d4a398a6508d9 (127.0.71.65:35227)
I20260929 01:58:28.744508 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:58:28.744728 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 6785
I20260929 01:58:28.746243 6847 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:58:28.825212 7317 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4db60feed0e84478ae90c5b868d0e44d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: true } } }
I20260929 01:58:28.825672 7317 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:28.828342 7553 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d: attempting to promote NON_VOTER 48ea3ceb72a14cda8d9b5a56c7bbfd3e to VOTER
I20260929 01:58:28.829428 7553 consensus_queue.cc:237] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: false } }
I20260929 01:58:28.832384 7371 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e [term 1 LEARNER]: Refusing update from remote peer 4db60feed0e84478ae90c5b868d0e44d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:58:28.832864 6979 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4 [term 1 FOLLOWER]: Refusing update from remote peer 4db60feed0e84478ae90c5b868d0e44d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:58:28.834270 7317 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [LEADER]: Connected to new peer: Peer: permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:58:28.836553 7317 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [LEADER]: Connected to new peer: Peer: permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:58:28.842437 7557 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 48ea3ceb72a14cda8d9b5a56c7bbfd3e (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: false } } }
I20260929 01:58:28.843715 7371 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 48ea3ceb72a14cda8d9b5a56c7bbfd3e (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: false } } }
I20260929 01:58:28.847565 7553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "4db60feed0e84478ae90c5b868d0e44d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: false } } }
I20260929 01:58:28.847856 7553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:28.847795 7399 sys_catalog.cc:455] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4db60feed0e84478ae90c5b868d0e44d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: false } } }
I20260929 01:58:28.848069 7399 sys_catalog.cc:458] T 00000000000000000000000000000000 P 48ea3ceb72a14cda8d9b5a56c7bbfd3e [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:28.848245 7553 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4db60feed0e84478ae90c5b868d0e44d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: false } } }
I20260929 01:58:28.848506 7553 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:28.848814 7557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4db60feed0e84478ae90c5b868d0e44d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: false } } }
I20260929 01:58:28.849001 7557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:28.849269 6979 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 48ea3ceb72a14cda8d9b5a56c7bbfd3e (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: false } } }
I20260929 01:58:28.853072 7559 sys_catalog.cc:455] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4db60feed0e84478ae90c5b868d0e44d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: false } } }
I20260929 01:58:28.853488 7559 sys_catalog.cc:458] T 00000000000000000000000000000000 P 789c69f6f65148bb96baf03e70e86fc4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:28.881829 6827 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [term 1 FOLLOWER]: Refusing update from remote peer 4db60feed0e84478ae90c5b868d0e44d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:58:28.882093 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 6785
I20260929 01:58:28.883370 7557 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [LEADER]: Connected to new peer: Peer: permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260929 01:58:28.909039 6880 connection.cc:573] client connection to 127.0.71.126:41987 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:58:28.910136 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:41987
--webserver_interface=127.0.71.126
--webserver_port=42025
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:36809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:41987,127.0.71.125:42953,127.0.71.124:44253
--master_addresses=127.0.71.126:41987,127.0.71.125:42953,127.0.71.124:44253,127.0.71.123:44065 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:58:28.911796 6880 consensus_peers.cc:597] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d -> Peer 2c3b21136a584b148e7f6be3709a4077 (127.0.71.126:41987): Couldn't send request to peer 2c3b21136a584b148e7f6be3709a4077. Status: Network error: Client connection negotiation failed: client connection to 127.0.71.126:41987: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260929 01:58:29.368161 7562 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:29.368748 7562 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:29.368901 7562 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:29.378731 7562 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:58:29.378929 7562 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:29.379067 7562 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:58:29.379155 7562 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:58:29.393767 7562 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:36809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:41987,127.0.71.125:42953,127.0.71.124:44253,127.0.71.123:44065
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:41987
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=42025
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:29.396130 7562 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:29.398008 7562 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:29.409157 7568 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:29.409562 7567 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:29.410562 7570 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:29.411234 7562 server_base.cc:1061] running on GCE node
I20260929 01:58:29.412307 7562 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:29.414162 7562 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:29.415496 7562 hybrid_clock.cc:648] HybridClock initialized: now 1790647109415460 us; error 76 us; skew 500 ppm
I20260929 01:58:29.416132 7562 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:29 dist-test-slave-vz23 krb5kdc[6769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:29.421238 7562 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:58:29.423789 7562 webserver.cc:533] Webserver started at http://127.0.71.126:42025/ using document root <none> and password file <none>
I20260929 01:58:29.425027 7562 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:29.425267 7562 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:29.433378 7562 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 01:58:29.437755 7577 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:29.439832 7562 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.005s sys 0.000s
I20260929 01:58:29.440127 7562 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "2c3b21136a584b148e7f6be3709a4077"
format_stamp: "Formatted at 2026-09-29 01:58:23 on dist-test-slave-vz23"
I20260929 01:58:29.441259 7562 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:29.498999 7562 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:29.512845 7562 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:29.513208 7562 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:29.540081 7562 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:41987
I20260929 01:58:29.540055 7628 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:41987 every 8 connection(s)
I20260929 01:58:29.542295 7562 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:58:29.550151 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 7562
I20260929 01:58:29.550523 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 7333
I20260929 01:58:29.551788 7629 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:58:29.557654 7629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077: Bootstrap starting.
I20260929 01:58:29.571167 7534 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:58:29.580106 7629 log.cc:826] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077: Log is configured to *not* fsync() on all Append() calls
I20260929 01:58:29.594901 7629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:58:29.595654 7629 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077: Bootstrap complete.
I20260929 01:58:29.604111 7629 raft_consensus.cc:359] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: true } }
I20260929 01:58:29.605225 7629 raft_consensus.cc:740] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 2c3b21136a584b148e7f6be3709a4077, State: Initialized, Role: FOLLOWER
I20260929 01:58:29.606742 7629 consensus_queue.cc:260] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: true } }
I20260929 01:58:29.607924 7635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: true } } }
I20260929 01:58:29.608433 7635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:29.609529 7629 sys_catalog.cc:565] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:58:29.633440 7646 catalog_manager.cc:1308] Loaded cluster ID: e10a6bfc0e2a4b2ba755de70bc579c20
I20260929 01:58:29.633580 7646 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077: loading cluster ID for follower catalog manager: success
I20260929 01:58:29.639103 7646 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077: acquiring CA information for follower catalog manager: success
I20260929 01:58:29.642894 7646 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Sep 29 01:58:29 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647109, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:58:29.703989 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 7333
I20260929 01:58:29.724820 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:35227
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:41987,127.0.71.125:42953,127.0.71.124:44253
--webserver_port=35241
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:36809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:41987,127.0.71.125:42953,127.0.71.124:44253,127.0.71.123:44065 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:58:29.948683 7604 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 48ea3ceb72a14cda8d9b5a56c7bbfd3e (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: false } } }
I20260929 01:58:29.960294 7656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "4db60feed0e84478ae90c5b868d0e44d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: false } } }
I20260929 01:58:29.960839 7656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:29.961262 7635 sys_catalog.cc:455] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 4db60feed0e84478ae90c5b868d0e44d. Latest consensus state: current_term: 1 leader_uuid: "4db60feed0e84478ae90c5b868d0e44d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: false } } }
I20260929 01:58:29.961521 7635 sys_catalog.cc:458] T 00000000000000000000000000000000 P 2c3b21136a584b148e7f6be3709a4077 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:58:30.175171 7653 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:30.175559 7653 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:30.175702 7653 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:30.185211 7653 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:30.185473 7653 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:58:30.198304 7653 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:36809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:35227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=35241
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:41987,127.0.71.125:42953,127.0.71.124:44253,127.0.71.123:44065
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:30.200167 7653 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:30.201972 7653 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:30.215281 7661 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:30.215445 7664 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:30.215281 7662 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:30.216377 7653 server_base.cc:1061] running on GCE node
I20260929 01:58:30.217193 7653 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:30.218596 7653 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:30.219925 7653 hybrid_clock.cc:648] HybridClock initialized: now 1790647110219856 us; error 69 us; skew 500 ppm
I20260929 01:58:30.220585 7653 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:30 dist-test-slave-vz23 krb5kdc[6769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:30.225019 7653 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:58:30.227192 7653 webserver.cc:533] Webserver started at http://127.0.71.65:35241/ using document root <none> and password file <none>
I20260929 01:58:30.228087 7653 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:30.228220 7653 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:30.237003 7653 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 01:58:30.241288 7671 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:30.243192 7653 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:58:30.243583 7653 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "faf5e50f688b449b825d4a398a6508d9"
format_stamp: "Formatted at 2026-09-29 01:58:26 on dist-test-slave-vz23"
I20260929 01:58:30.244649 7653 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:30.282189 7653 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:30.287302 7653 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:30.287678 7653 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:30.289049 7653 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:58:30.291198 7653 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:58:30.291391 7653 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:30.291580 7653 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:58:30.291687 7653 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:30.337796 7653 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:35227
I20260929 01:58:30.337874 7783 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:35227 every 8 connection(s)
I20260929 01:58:30.339879 7653 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:58:30.346935 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 7653
Sep 29 01:58:30 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:58:30 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:58:30.379324 7786 heartbeater.cc:344] Connected to a master server at 127.0.71.126:41987
I20260929 01:58:30.379756 7786 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:30.380250 7785 heartbeater.cc:344] Connected to a master server at 127.0.71.124:44253
I20260929 01:58:30.380486 7785 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:30.381371 7786 heartbeater.cc:507] Master 127.0.71.126:41987 requested a full tablet report, sending...
I20260929 01:58:30.382293 7785 heartbeater.cc:507] Master 127.0.71.124:44253 requested a full tablet report, sending...
Sep 29 01:58:30 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:58:30.384780 7594 ts_manager.cc:194] Registered new tserver with Master: faf5e50f688b449b825d4a398a6508d9 (127.0.71.65:35227)
I20260929 01:58:30.386612 6969 ts_manager.cc:194] Re-registered known tserver with Master: faf5e50f688b449b825d4a398a6508d9 (127.0.71.65:35227)
Sep 29 01:58:30 dist-test-slave-vz23 krb5kdc[6769](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
I20260929 01:58:30.396533 7787 heartbeater.cc:344] Connected to a master server at 127.0.71.125:42953
I20260929 01:58:30.396790 7787 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:30.397545 7787 heartbeater.cc:507] Master 127.0.71.125:42953 requested a full tablet report, sending...
I20260929 01:58:30.398623 7784 heartbeater.cc:344] Connected to a master server at 127.0.71.123:44065
I20260929 01:58:30.398847 7784 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:30.399223 6893 ts_manager.cc:194] Re-registered known tserver with Master: faf5e50f688b449b825d4a398a6508d9 (127.0.71.65:35227)
I20260929 01:58:30.399469 7784 heartbeater.cc:507] Master 127.0.71.123:44065 requested a full tablet report, sending...
I20260929 01:58:30.400434 6893 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:59775
I20260929 01:58:30.401531 7360 ts_manager.cc:194] Re-registered known tserver with Master: faf5e50f688b449b825d4a398a6508d9 (127.0.71.65:35227)
I20260929 01:58:30.437418 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 7653
W20260929 01:58:30.438802 7799 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:58:30.434577 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:54710 (local address 127.0.71.65:35227)
0929 01:58:30.435269 (+ 692us) server_negotiation.cc:207] Beginning negotiation
0929 01:58:30.435278 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0929 01:58:30.435340 (+ 62us) server_negotiation.cc:408] Connection header received
0929 01:58:30.435412 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:58:30.435421 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:58:30.435802 (+ 381us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:58:30.436010 (+ 208us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:58:30.437221 (+ 1211us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:54710: BlockingRecv error: recv got EOF from 127.0.0.1:54710 (error 108)
Metrics: {"server-negotiator.queue_time_us":521,"thread_start_us":358,"threads_started":1}
W20260929 01:58:30.440639 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 7653: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:35227: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:58:30.440941 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 7653
I20260929 01:58:30.461597 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:35227
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:41987,127.0.71.125:42953,127.0.71.124:44253
--webserver_port=35241
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:36809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:41987,127.0.71.125:42953,127.0.71.124:44253,127.0.71.123:44065
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:58:30.480406 7557 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "4db60feed0e84478ae90c5b868d0e44d" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "2c3b21136a584b148e7f6be3709a4077" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41987 } } peers { permanent_uuid: "4db60feed0e84478ae90c5b868d0e44d" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 42953 } } peers { permanent_uuid: "789c69f6f65148bb96baf03e70e86fc4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 44253 } } peers { permanent_uuid: "48ea3ceb72a14cda8d9b5a56c7bbfd3e" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44065 } attrs { promote: false } } }
I20260929 01:58:30.480895 7557 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4db60feed0e84478ae90c5b868d0e44d [sys.catalog]: This master's current role is: LEADER
W20260929 01:58:30.883855 7801 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:30.884243 7801 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:30.884433 7801 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:58:30.884603 7801 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:30.894232 7801 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:30.894562 7801 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:58:30.909935 7801 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:36809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:35227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=35241
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:41987,127.0.71.125:42953,127.0.71.124:44253,127.0.71.123:44065
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:30.911978 7801 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:30.914161 7801 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:30.927556 7809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:30.927584 7810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:30.928560 7812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:30.929468 7801 server_base.cc:1061] running on GCE node
I20260929 01:58:30.930225 7801 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:30.931535 7801 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:30.933032 7801 hybrid_clock.cc:648] HybridClock initialized: now 1790647110932916 us; error 71 us; skew 500 ppm
I20260929 01:58:30.933589 7801 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:30 dist-test-slave-vz23 krb5kdc[6769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647110, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:30.938145 7801 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:58:30.940737 7801 webserver.cc:533] Webserver started at http://127.0.71.65:35241/ using document root <none> and password file <none>
I20260929 01:58:30.941984 7801 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:30.942142 7801 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:30.950353 7801 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.002s
I20260929 01:58:30.955143 7819 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:30.957594 7801 fs_manager.cc:730] Time spent opening block manager: real 0.006s user 0.003s sys 0.003s
I20260929 01:58:30.957917 7801 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "faf5e50f688b449b825d4a398a6508d9"
format_stamp: "Formatted at 2026-09-29 01:58:26 on dist-test-slave-vz23"
I20260929 01:58:30.958961 7801 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:31.016149 7801 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:31.022145 7801 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:31.022570 7801 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:31.023829 7801 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:58:31.026105 7801 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:58:31.026242 7801 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:31.026381 7801 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:58:31.026448 7801 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:58:31.027050 7801 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:58:31.027154 7801 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:58:31.027287 7801 fault_injection.cc:59] injecting fault for kudu (pid 7801): FLAGS_fault_before_start (process will exit)
I20260929 01:58:31.049783 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:35227
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:41987,127.0.71.125:42953,127.0.71.124:44253
--webserver_port=35241
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:36809
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:41987,127.0.71.125:42953,127.0.71.124:44253,127.0.71.123:44065 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:58:31.477550 7827 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:31.477931 7827 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:31.478114 7827 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:31.487625 7827 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:31.487934 7827 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:58:31.500741 7827 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:36809
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:35227
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=35241
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:41987,127.0.71.125:42953,127.0.71.124:44253,127.0.71.123:44065
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:31.502871 7827 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:31.504678 7827 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:31.518510 7832 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:31.518510 7833 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:31.518996 7835 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:31.519994 7827 server_base.cc:1061] running on GCE node
I20260929 01:58:31.520957 7827 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:31.522307 7827 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:31.523576 7827 hybrid_clock.cc:648] HybridClock initialized: now 1790647111523526 us; error 64 us; skew 500 ppm
I20260929 01:58:31.524027 7827 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:31 dist-test-slave-vz23 krb5kdc[6769](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647111, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:31.528828 7827 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:58:31.531272 7827 webserver.cc:533] Webserver started at http://127.0.71.65:35241/ using document root <none> and password file <none>
I20260929 01:58:31.532171 7827 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:31.532313 7827 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:31.539799 7827 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:58:31.544404 7842 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:31.546629 7827 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.001s sys 0.002s
I20260929 01:58:31.546973 7827 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "faf5e50f688b449b825d4a398a6508d9"
format_stamp: "Formatted at 2026-09-29 01:58:26 on dist-test-slave-vz23"
I20260929 01:58:31.548136 7827 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:31.572604 7827 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:31.578069 7827 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:31.578429 7827 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:31.579991 7827 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:58:31.582142 7827 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:58:31.582283 7827 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:31.582414 7827 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:58:31.582506 7827 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:31.625561 7827 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:35227
I20260929 01:58:31.625661 7954 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:35227 every 8 connection(s)
I20260929 01:58:31.627580 7827 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:58:31.636900 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 7827
I20260929 01:58:31.637281 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 7827
W20260929 01:58:31.657159 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 7827: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:35227: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:58:31.657425 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 7827
I20260929 01:58:31.680881 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 7562
W20260929 01:58:31.681802 7960 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:58:31.632009 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:41693 (local address 127.0.71.126:41987)
0929 01:58:31.632763 (+ 754us) server_negotiation.cc:207] Beginning negotiation
0929 01:58:31.632774 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0929 01:58:31.635160 (+ 2386us) server_negotiation.cc:408] Connection header received
0929 01:58:31.638189 (+ 3029us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:58:31.638198 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:58:31.641650 (+ 3452us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:58:31.641886 (+ 236us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:58:31.655976 (+ 14090us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:58:31.657976 (+ 2000us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:58:31.680715 (+ 22739us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:41693: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":526,"thread_start_us":319,"threads_started":1}
W20260929 01:58:31.681794 7967 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:58:31.645798 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:47849 (local address 127.0.71.124:44253)
0929 01:58:31.658803 (+ 13005us) server_negotiation.cc:207] Beginning negotiation
0929 01:58:31.658812 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0929 01:58:31.680697 (+ 21885us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:47849: BlockingRecv error: recv got EOF from 127.0.71.65:47849 (error 108)
Metrics: {"server-negotiator.queue_time_us":12807,"thread_start_us":13010,"threads_started":1}
W20260929 01:58:31.681828 7963 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:58:31.635748 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:39643 (local address 127.0.71.123:44065)
0929 01:58:31.654001 (+ 18253us) server_negotiation.cc:207] Beginning negotiation
0929 01:58:31.654012 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0929 01:58:31.654045 (+ 33us) server_negotiation.cc:408] Connection header received
0929 01:58:31.654151 (+ 106us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:58:31.654161 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:58:31.654315 (+ 154us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:58:31.654541 (+ 226us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:58:31.660149 (+ 5608us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:58:31.661933 (+ 1784us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:58:31.680684 (+ 18751us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:39643: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":18039,"thread_start_us":18999,"threads_started":1}
W20260929 01:58:31.683988 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 7562: Not authorized: Client connection negotiation failed: client connection to 127.0.71.126:41987: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:58:31.684171 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 7562
W20260929 01:58:31.684195 7960 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:58:31.682495 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:59216 (local address 127.0.71.126:41987)
0929 01:58:31.682697 (+ 202us) server_negotiation.cc:207] Beginning negotiation
0929 01:58:31.682702 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0929 01:58:31.682721 (+ 19us) server_negotiation.cc:408] Connection header received
0929 01:58:31.682793 (+ 72us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:58:31.682797 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:58:31.682869 (+ 72us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:58:31.682970 (+ 101us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:58:31.683883 (+ 913us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:59216: BlockingRecv error: recv got EOF from 127.0.0.1:59216 (error 108)
Metrics: {"server-negotiator.queue_time_us":78}
I20260929 01:58:31.710731 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 6861
I20260929 01:58:31.712574 6923 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:58:31.844787 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 6861
I20260929 01:58:31.870857 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 6938
I20260929 01:58:31.872290 7000 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:58:31.994657 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 6938
I20260929 01:58:32.019970 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 7216
I20260929 01:58:32.021564 7391 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:58:32.147502 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 7216
2026-09-29T01:58:32Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestAddMaster/7 (11394 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/8
Loading random data
Initializing database '/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:58:32 dist-test-slave-vz23 krb5kdc[7973](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:58:32 dist-test-slave-vz23 krb5kdc[7973](info): set up 2 sockets
Sep 29 01:58:32 dist-test-slave-vz23 krb5kdc[7973](info): commencing operation
krb5kdc: starting...
W20260929 01:58:34.366595 285 mini_kdc.cc:121] Time spent starting KDC: real 2.170s user 0.004s sys 0.031s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 01:58:34 dist-test-slave-vz23 krb5kdc[7973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:58:34Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:58:34Z Disabled control of system clock
2026-09-29T01:58:34Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:58:34Z Disabled control of system clock
2026-09-29T01:58:34Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:58:34Z Disabled control of system clock
2026-09-29T01:58:34Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:58:34Z Disabled control of system clock
2026-09-29T01:58:34Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:58:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:58:34.950131 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:41357
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:34213,127.0.71.85:33543,127.0.71.86:41399,127.0.71.87:42679,127.0.71.88:37589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:41357,127.0.71.125:39313,127.0.71.124:45665 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:58:35.377039 7997 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:35.377502 7997 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:35.377679 7997 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:35.388106 7997 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:58:35.388264 7997 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:35.388379 7997 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:58:35.388525 7997 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:58:35.402290 7997 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:34213,127.0.71.85:33543,127.0.71.86:41399,127.0.71.87:42679,127.0.71.88:37589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:41357,127.0.71.125:39313,127.0.71.124:45665
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:41357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:35.404320 7997 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:35.406096 7997 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:35.417364 8002 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:35.417605 8005 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:35.417397 8003 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:35.419386 7997 server_base.cc:1061] running on GCE node
I20260929 01:58:35.420619 7997 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:35.422801 7997 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:35.426311 7997 hybrid_clock.cc:648] HybridClock initialized: now 1790647115426078 us; error 153 us; skew 500 ppm
I20260929 01:58:35.426889 7997 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:35 dist-test-slave-vz23 krb5kdc[7973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:35.431464 7997 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:58:35.433832 7997 webserver.cc:533] Webserver started at http://127.0.71.126:39613/ using document root <none> and password file <none>
I20260929 01:58:35.434768 7997 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:35.434947 7997 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:35.435389 7997 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:58:35.438628 7997 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "5cc926876156403ba1430e293b6c57f5"
format_stamp: "Formatted at 2026-09-29 01:58:35 on dist-test-slave-vz23"
I20260929 01:58:35.439968 7997 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "5cc926876156403ba1430e293b6c57f5"
format_stamp: "Formatted at 2026-09-29 01:58:35 on dist-test-slave-vz23"
I20260929 01:58:35.447368 7997 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260929 01:58:35.452416 8012 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:35.454376 7997 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:58:35.454684 7997 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "5cc926876156403ba1430e293b6c57f5"
format_stamp: "Formatted at 2026-09-29 01:58:35 on dist-test-slave-vz23"
I20260929 01:58:35.454988 7997 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:35.482020 7997 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:35.493808 7997 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:35.494218 7997 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:35.520961 7997 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:41357
I20260929 01:58:35.520992 8063 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:41357 every 8 connection(s)
I20260929 01:58:35.522960 7997 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:58:35.525969 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 7997
I20260929 01:58:35.526232 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:58:35.529031 8064 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:58:35.536280 8064 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:58:35 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.125@KRBTEST.COM" created.
I20260929 01:58:35.569041 8064 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:58:35.571179 8016 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.125:39313: connect: Connection refused (error 111)
W20260929 01:58:35.573581 8064 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:39313: Network error: Client connection negotiation failed: client connection to 127.0.71.125:39313: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:58:35.623131 8064 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } attempt: 1
W20260929 01:58:35.626152 8064 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:39313: Network error: Client connection negotiation failed: client connection to 127.0.71.125:39313: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.0.71.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:58:35.668857 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:39313
--webserver_interface=127.0.71.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:34213,127.0.71.85:33543,127.0.71.86:41399,127.0.71.87:42679,127.0.71.88:37589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:41357,127.0.71.125:39313,127.0.71.124:45665 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:58:35.694476 8064 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } attempt: 2
W20260929 01:58:35.697330 8064 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:39313: Network error: Client connection negotiation failed: client connection to 127.0.71.125:39313: connect: Connection refused (error 111)
I20260929 01:58:35.788700 8064 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } attempt: 3
W20260929 01:58:35.791692 8064 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:39313: Network error: Client connection negotiation failed: client connection to 127.0.71.125:39313: connect: Connection refused (error 111)
I20260929 01:58:35.935066 8064 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } attempt: 4
W20260929 01:58:35.938077 8064 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:39313: Network error: Client connection negotiation failed: client connection to 127.0.71.125:39313: connect: Connection refused (error 111)
W20260929 01:58:36.080556 8073 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:36.080991 8073 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:36.081192 8073 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:36.096160 8073 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:58:36.096315 8073 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:36.096481 8073 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:58:36.096590 8073 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:58:36.109946 8073 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:34213,127.0.71.85:33543,127.0.71.86:41399,127.0.71.87:42679,127.0.71.88:37589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:41357,127.0.71.125:39313,127.0.71.124:45665
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:39313
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:36.111837 8073 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:36.113659 8073 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:36.124384 8081 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:36.124493 8079 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:36.124534 8078 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:36.131376 8073 server_base.cc:1061] running on GCE node
I20260929 01:58:36.132537 8073 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:36.134579 8073 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:36.136029 8073 hybrid_clock.cc:648] HybridClock initialized: now 1790647116135874 us; error 105 us; skew 500 ppm
I20260929 01:58:36.136600 8073 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:36 dist-test-slave-vz23 krb5kdc[7973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:36.140717 8073 init.cc:377] Logged in from keytab as kudu/127.0.71.125@KRBTEST.COM (short username kudu)
I20260929 01:58:36.143082 8073 webserver.cc:533] Webserver started at http://127.0.71.125:37175/ using document root <none> and password file <none>
I20260929 01:58:36.143932 8073 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:36.144057 8073 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:36.144433 8073 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:58:36.147503 8073 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-1/data/instance:
uuid: "24103e5cfa7e4e0d9788bfe029d7a44a"
format_stamp: "Formatted at 2026-09-29 01:58:36 on dist-test-slave-vz23"
I20260929 01:58:36.148558 8073 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-1/wal/instance:
uuid: "24103e5cfa7e4e0d9788bfe029d7a44a"
format_stamp: "Formatted at 2026-09-29 01:58:36 on dist-test-slave-vz23"
I20260929 01:58:36.155539 8073 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.005s sys 0.000s
I20260929 01:58:36.160322 8088 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:36.162564 8073 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 01:58:36.162796 8073 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "24103e5cfa7e4e0d9788bfe029d7a44a"
format_stamp: "Formatted at 2026-09-29 01:58:36 on dist-test-slave-vz23"
I20260929 01:58:36.163008 8073 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:36.185518 8073 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:36.190449 8073 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:36.190740 8073 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:36.215663 8073 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:39313
I20260929 01:58:36.215688 8139 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:39313 every 8 connection(s)
I20260929 01:58:36.217901 8073 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:58:36.223553 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 8073
I20260929 01:58:36.223649 8140 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:58:36.223811 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:58:36.234218 8140 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:58:36.237413 8064 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } attempt: 5
Sep 29 01:58:36 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:58:36 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.124@KRBTEST.COM; defaulting to no policy
I20260929 01:58:36.265697 8140 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.124@KRBTEST.COM" created.
I20260929 01:58:36.267536 8064 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:58:36.270104 8064 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:45665: Network error: Client connection negotiation failed: client connection to 127.0.71.124:45665: connect: Connection refused (error 111)
Sep 29 01:58:36 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:58:36.287396 8140 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:58:36.288789 8092 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.124:45665: connect: Connection refused (error 111)
W20260929 01:58:36.291054 8140 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:45665: Network error: Client connection negotiation failed: client connection to 127.0.71.124:45665: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:58:36.321408 8064 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } attempt: 1
W20260929 01:58:36.324016 8064 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:45665: Network error: Client connection negotiation failed: client connection to 127.0.71.124:45665: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.0.71.124@KRBTEST.COM; defaulting to no policy
I20260929 01:58:36.340905 8140 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } attempt: 1
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.124@KRBTEST.COM" created.
W20260929 01:58:36.344336 8140 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:45665: Network error: Client connection negotiation failed: client connection to 127.0.71.124:45665: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:58:36.386551 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:45665
--webserver_interface=127.0.71.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:34213,127.0.71.85:33543,127.0.71.86:41399,127.0.71.87:42679,127.0.71.88:37589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:41357,127.0.71.125:39313,127.0.71.124:45665 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:58:36.392390 8064 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } attempt: 2
W20260929 01:58:36.395351 8064 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:45665: Network error: Client connection negotiation failed: client connection to 127.0.71.124:45665: connect: Connection refused (error 111)
I20260929 01:58:36.412815 8140 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } attempt: 2
W20260929 01:58:36.415500 8140 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:45665: Network error: Client connection negotiation failed: client connection to 127.0.71.124:45665: connect: Connection refused (error 111)
I20260929 01:58:36.501677 8064 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } attempt: 3
W20260929 01:58:36.504701 8064 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:45665: Network error: Client connection negotiation failed: client connection to 127.0.71.124:45665: connect: Connection refused (error 111)
I20260929 01:58:36.506830 8140 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } attempt: 3
W20260929 01:58:36.509428 8140 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:45665: Network error: Client connection negotiation failed: client connection to 127.0.71.124:45665: connect: Connection refused (error 111)
I20260929 01:58:36.652766 8140 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } attempt: 4
W20260929 01:58:36.655521 8140 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:45665: Network error: Client connection negotiation failed: client connection to 127.0.71.124:45665: connect: Connection refused (error 111)
I20260929 01:58:36.682055 8064 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } attempt: 4
W20260929 01:58:36.684849 8064 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:45665: Network error: Client connection negotiation failed: client connection to 127.0.71.124:45665: connect: Connection refused (error 111)
W20260929 01:58:36.790002 8150 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:36.790408 8150 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:36.790501 8150 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:36.799973 8150 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:58:36.800097 8150 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:36.800156 8150 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:58:36.800204 8150 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:58:36.812633 8150 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:34213,127.0.71.85:33543,127.0.71.86:41399,127.0.71.87:42679,127.0.71.88:37589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:41357,127.0.71.125:39313,127.0.71.124:45665
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:45665
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:36.814841 8150 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:36.816794 8150 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:36.829152 8156 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:36.829102 8155 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:36.829653 8158 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:36.831737 8150 server_base.cc:1061] running on GCE node
I20260929 01:58:36.832976 8150 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:36.835256 8150 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:36.838904 8150 hybrid_clock.cc:648] HybridClock initialized: now 1790647116838579 us; error 200 us; skew 500 ppm
I20260929 01:58:36.839514 8150 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:36 dist-test-slave-vz23 krb5kdc[7973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:36.844367 8150 init.cc:377] Logged in from keytab as kudu/127.0.71.124@KRBTEST.COM (short username kudu)
I20260929 01:58:36.847013 8150 webserver.cc:533] Webserver started at http://127.0.71.124:46257/ using document root <none> and password file <none>
I20260929 01:58:36.848095 8150 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:36.848299 8150 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:36.848783 8150 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:58:36.852156 8150 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-2/data/instance:
uuid: "070f469f2361414ebb8f17e7ba40ddff"
format_stamp: "Formatted at 2026-09-29 01:58:36 on dist-test-slave-vz23"
I20260929 01:58:36.853369 8150 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-2/wal/instance:
uuid: "070f469f2361414ebb8f17e7ba40ddff"
format_stamp: "Formatted at 2026-09-29 01:58:36 on dist-test-slave-vz23"
I20260929 01:58:36.861133 8150 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260929 01:58:36.866317 8165 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:36.868702 8150 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:58:36.869017 8150 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "070f469f2361414ebb8f17e7ba40ddff"
format_stamp: "Formatted at 2026-09-29 01:58:36 on dist-test-slave-vz23"
I20260929 01:58:36.869293 8150 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:36.894621 8150 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:36.899003 8150 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:36.899351 8150 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:36.922909 8150 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:45665
I20260929 01:58:36.922943 8216 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:45665 every 8 connection(s)
I20260929 01:58:36.925133 8150 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:58:36.930884 8217 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:58:36.931473 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 8150
I20260929 01:58:36.931700 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-2/wal/instance
Sep 29 01:58:36 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:58:36.946471 8217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:58:36.954825 8140 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } attempt: 5
I20260929 01:58:36.962116 8064 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } attempt: 5
Sep 29 01:58:36 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:58:36 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:58:36 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:58:36.984002 8217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:58:36 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647115, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:58:36.996243 8140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a: Bootstrap starting.
Sep 29 01:58:36 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:58:36 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:58:37.004240 8140 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a: Neither blocks nor log segments found. Creating new log.
I20260929 01:58:37.006765 8140 log.cc:826] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a: Log is configured to *not* fsync() on all Append() calls
I20260929 01:58:37.012964 8217 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:58:37.013346 8064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5: Bootstrap starting.
I20260929 01:58:37.015638 8140 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a: No bootstrap required, opened a new log
I20260929 01:58:37.019183 8064 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5: Neither blocks nor log segments found. Creating new log.
I20260929 01:58:37.021070 8064 log.cc:826] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5: Log is configured to *not* fsync() on all Append() calls
Sep 29 01:58:37 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:58:37.025380 8140 raft_consensus.cc:359] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } }
I20260929 01:58:37.026041 8140 raft_consensus.cc:385] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:58:37.026213 8064 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5: No bootstrap required, opened a new log
I20260929 01:58:37.026265 8140 raft_consensus.cc:740] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 24103e5cfa7e4e0d9788bfe029d7a44a, State: Initialized, Role: FOLLOWER
I20260929 01:58:37.027426 8140 consensus_queue.cc:260] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } }
I20260929 01:58:37.028963 8229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } }
I20260929 01:58:37.029577 8229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:37.030701 8140 sys_catalog.cc:565] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:58:37.038781 8064 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } }
I20260929 01:58:37.039217 8064 raft_consensus.cc:385] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:58:37.039405 8064 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cc926876156403ba1430e293b6c57f5, State: Initialized, Role: FOLLOWER
I20260929 01:58:37.040385 8064 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } }
I20260929 01:58:37.041841 8217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff: Bootstrap starting.
I20260929 01:58:37.043642 8236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } }
I20260929 01:58:37.044155 8236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:37.045342 8064 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:58:37.049314 8217 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff: Neither blocks nor log segments found. Creating new log.
I20260929 01:58:37.051904 8217 log.cc:826] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff: Log is configured to *not* fsync() on all Append() calls
I20260929 01:58:37.057950 8217 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff: No bootstrap required, opened a new log
I20260929 01:58:37.067716 8217 raft_consensus.cc:359] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } }
I20260929 01:58:37.068390 8217 raft_consensus.cc:385] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:58:37.068632 8217 raft_consensus.cc:740] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 070f469f2361414ebb8f17e7ba40ddff, State: Initialized, Role: FOLLOWER
I20260929 01:58:37.069693 8217 consensus_queue.cc:260] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } }
I20260929 01:58:37.061316 8073 master_runner.cc:187] Error getting master registration for 127.0.71.126:41357: OK, instance_id { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" instance_seqno: 1790647115513885 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:58:37.071391 8073 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:58:37.076807 8247 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
I20260929 01:58:37.076679 8253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } }
I20260929 01:58:37.077100 8253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:58:37.077330 8247 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:58:37.078518 8217 sys_catalog.cc:565] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:58:37.100544 7997 master_runner.cc:187] Error getting master registration for 127.0.71.124:45665: OK, instance_id { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" instance_seqno: 1790647116916537 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:58:37.129081 7997 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:58:37.129479 8255 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:58:37.129720 8255 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 01:58:37.151921 8171 tablet.cc:2433] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:58:37.159262 8150 master_runner.cc:264] No leader master found from master 070f469f2361414ebb8f17e7ba40ddff
I20260929 01:58:37.159716 8150 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:58:37.168063 8267 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:58:37.168505 8267 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 01:58:37.194334 8094 tablet.cc:2433] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:58:37.195995 8253 raft_consensus.cc:493] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:58:37.196376 8253 raft_consensus.cc:515] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } }
I20260929 01:58:37.198788 8253 leader_election.cc:290] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 5cc926876156403ba1430e293b6c57f5 (127.0.71.126:41357), 24103e5cfa7e4e0d9788bfe029d7a44a (127.0.71.125:39313)
I20260929 01:58:37.200845 8115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "070f469f2361414ebb8f17e7ba40ddff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" is_pre_election: true
I20260929 01:58:37.201583 8115 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 070f469f2361414ebb8f17e7ba40ddff in term 0.
I20260929 01:58:37.202812 8169 leader_election.cc:304] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 070f469f2361414ebb8f17e7ba40ddff, 24103e5cfa7e4e0d9788bfe029d7a44a; no voters:
I20260929 01:58:37.206336 8039 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "070f469f2361414ebb8f17e7ba40ddff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cc926876156403ba1430e293b6c57f5" is_pre_election: true
I20260929 01:58:37.207082 8039 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 070f469f2361414ebb8f17e7ba40ddff in term 0.
I20260929 01:58:37.207146 8253 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:58:37.207549 8253 raft_consensus.cc:493] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:58:37.208617 8253 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:58:37.213070 8253 raft_consensus.cc:515] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } }
I20260929 01:58:37.215107 8039 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "070f469f2361414ebb8f17e7ba40ddff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "5cc926876156403ba1430e293b6c57f5"
I20260929 01:58:37.215457 8039 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:58:37.220386 8039 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 070f469f2361414ebb8f17e7ba40ddff in term 1.
I20260929 01:58:37.226886 8166 leader_election.cc:304] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 070f469f2361414ebb8f17e7ba40ddff, 5cc926876156403ba1430e293b6c57f5; no voters:
I20260929 01:58:37.233515 8115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "070f469f2361414ebb8f17e7ba40ddff" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a"
I20260929 01:58:37.233894 8115 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:58:37.238541 8115 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 070f469f2361414ebb8f17e7ba40ddff in term 1.
I20260929 01:58:37.239868 8253 leader_election.cc:290] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [CANDIDATE]: Term 1 election: Requested vote from peers 5cc926876156403ba1430e293b6c57f5 (127.0.71.126:41357), 24103e5cfa7e4e0d9788bfe029d7a44a (127.0.71.125:39313)
I20260929 01:58:37.240604 8253 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:58:37.241313 8253 raft_consensus.cc:697] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [term 1 LEADER]: Becoming Leader. State: Replica: 070f469f2361414ebb8f17e7ba40ddff, State: Running, Role: LEADER
I20260929 01:58:37.242406 8253 consensus_queue.cc:237] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } }
W20260929 01:58:37.248943 8018 tablet.cc:2433] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:58:37.266781 8269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: SysCatalogTable state changed. Reason: New leader 070f469f2361414ebb8f17e7ba40ddff. Latest consensus state: current_term: 1 leader_uuid: "070f469f2361414ebb8f17e7ba40ddff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } }
I20260929 01:58:37.267297 8269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:37.276788 8271 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:58:37.279985 8271 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:58:37.304797 8115 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a [term 1 FOLLOWER]: Refusing update from remote peer 070f469f2361414ebb8f17e7ba40ddff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:58:37.306523 8269 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [LEADER]: Connected to new peer: Peer: permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:58:37.321313 8039 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [term 1 FOLLOWER]: Refusing update from remote peer 070f469f2361414ebb8f17e7ba40ddff: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:58:37.323753 8269 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:58:37.354411 8236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 070f469f2361414ebb8f17e7ba40ddff. Latest consensus state: current_term: 1 leader_uuid: "070f469f2361414ebb8f17e7ba40ddff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } }
I20260929 01:58:37.354794 8236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:37.362171 8272 mvcc.cc:204] Tried to move back new op lower bound from 7334490592459001856 to 7334490592267149312. Current Snapshot: MvccSnapshot[applied={T|T < 7334490592459001856}]
I20260929 01:58:37.363271 8236 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "070f469f2361414ebb8f17e7ba40ddff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } }
I20260929 01:58:37.363567 8236 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:37.373175 8253 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "070f469f2361414ebb8f17e7ba40ddff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } }
I20260929 01:58:37.373605 8253 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:37.383592 8274 mvcc.cc:204] Tried to move back new op lower bound from 7334490592459001856 to 7334490592267149312. Current Snapshot: MvccSnapshot[applied={T|T < 7334490592459001856}]
I20260929 01:58:37.389073 8271 catalog_manager.cc:1396] Generated new cluster ID: 66a1b2eb2961442d98965e9214955dcc
I20260929 01:58:37.389331 8271 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:58:37.411201 8229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a [sys.catalog]: SysCatalogTable state changed. Reason: New leader 070f469f2361414ebb8f17e7ba40ddff. Latest consensus state: current_term: 1 leader_uuid: "070f469f2361414ebb8f17e7ba40ddff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } }
I20260929 01:58:37.411638 8229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:37.416025 8269 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "070f469f2361414ebb8f17e7ba40ddff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } }
I20260929 01:58:37.416762 8269 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:37.423209 8273 mvcc.cc:204] Tried to move back new op lower bound from 7334490592459001856 to 7334490592267149312. Current Snapshot: MvccSnapshot[applied={T|T < 7334490592459001856}]
I20260929 01:58:37.447572 8271 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 01:58:37.451222 8271 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:58:37.470896 8229 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "070f469f2361414ebb8f17e7ba40ddff" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } }
I20260929 01:58:37.471274 8229 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:37.487617 8271 catalog_manager.cc:6219] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff: Generated new TSK 0
I20260929 01:58:37.490489 8271 catalog_manager.cc:1563] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
W20260929 01:58:37.645861 285 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.0.71.65: real 0.119s user 0.002s sys 0.029s
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:58:37.745596 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:41357,127.0.71.125:39313,127.0.71.124:45665
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:34213,127.0.71.85:33543,127.0.71.86:41399,127.0.71.87:42679,127.0.71.88:37589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:58:38.080291 8247 catalog_manager.cc:1308] Loaded cluster ID: 66a1b2eb2961442d98965e9214955dcc
I20260929 01:58:38.080510 8247 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a: loading cluster ID for follower catalog manager: success
I20260929 01:58:38.085008 8247 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a: acquiring CA information for follower catalog manager: success
I20260929 01:58:38.088435 8247 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:58:38.132791 8255 catalog_manager.cc:1308] Loaded cluster ID: 66a1b2eb2961442d98965e9214955dcc
I20260929 01:58:38.132948 8255 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5: loading cluster ID for follower catalog manager: success
I20260929 01:58:38.137796 8255 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5: acquiring CA information for follower catalog manager: success
I20260929 01:58:38.141189 8255 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 01:58:38.238067 8285 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:38.238497 8285 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:38.238612 8285 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:38.248263 8285 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:38.248637 8285 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:58:38.261264 8285 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:34213,127.0.71.85:33543,127.0.71.86:41399,127.0.71.87:42679,127.0.71.88:37589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:41357,127.0.71.125:39313,127.0.71.124:45665
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:38.263088 8285 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:38.264778 8285 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:38.276628 8291 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:38.277743 8292 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:38.279633 8294 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:38.279891 8285 server_base.cc:1061] running on GCE node
I20260929 01:58:38.280974 8285 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:38.282603 8285 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:38.286113 8285 hybrid_clock.cc:648] HybridClock initialized: now 1790647118285877 us; error 143 us; skew 500 ppm
I20260929 01:58:38.286767 8285 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:38 dist-test-slave-vz23 krb5kdc[7973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:38.291062 8285 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:58:38.293864 8285 webserver.cc:533] Webserver started at http://127.0.71.65:46393/ using document root <none> and password file <none>
I20260929 01:58:38.294821 8285 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:38.295022 8285 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:38.295467 8285 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:58:38.298729 8285 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "e1fcd1a5f6284d28b5f9f6eba530f0ba"
format_stamp: "Formatted at 2026-09-29 01:58:38 on dist-test-slave-vz23"
I20260929 01:58:38.299741 8285 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "e1fcd1a5f6284d28b5f9f6eba530f0ba"
format_stamp: "Formatted at 2026-09-29 01:58:38 on dist-test-slave-vz23"
I20260929 01:58:38.307631 8285 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.009s sys 0.000s
I20260929 01:58:38.312769 8301 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:38.314791 8285 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:58:38.315120 8285 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "e1fcd1a5f6284d28b5f9f6eba530f0ba"
format_stamp: "Formatted at 2026-09-29 01:58:38 on dist-test-slave-vz23"
I20260929 01:58:38.315392 8285 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:38.332640 8285 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:38.338093 8285 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:38.338490 8285 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:38.339709 8285 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:58:38.342054 8285 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:58:38.342201 8285 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:38.342365 8285 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:58:38.342438 8285 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:38.383759 8285 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:38181
I20260929 01:58:38.383833 8413 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:38181 every 8 connection(s)
I20260929 01:58:38.385537 8285 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:58:38.388150 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 8285
I20260929 01:58:38.388381 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
Sep 29 01:58:38 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:58:38 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:58:38 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647118, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:58:38.421541 8415 heartbeater.cc:344] Connected to a master server at 127.0.71.126:41357
I20260929 01:58:38.422048 8415 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:38.423501 8415 heartbeater.cc:507] Master 127.0.71.126:41357 requested a full tablet report, sending...
I20260929 01:58:38.424801 8414 heartbeater.cc:344] Connected to a master server at 127.0.71.124:45665
I20260929 01:58:38.424983 8414 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:38.425647 8414 heartbeater.cc:507] Master 127.0.71.124:45665 requested a full tablet report, sending...
I20260929 01:58:38.426963 8028 ts_manager.cc:194] Registered new tserver with Master: e1fcd1a5f6284d28b5f9f6eba530f0ba (127.0.71.65:38181)
I20260929 01:58:38.427868 8182 ts_manager.cc:194] Registered new tserver with Master: e1fcd1a5f6284d28b5f9f6eba530f0ba (127.0.71.65:38181)
I20260929 01:58:38.428313 8416 heartbeater.cc:344] Connected to a master server at 127.0.71.125:39313
I20260929 01:58:38.428520 8416 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:38.429147 8416 heartbeater.cc:507] Master 127.0.71.125:39313 requested a full tablet report, sending...
I20260929 01:58:38.430289 8182 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:58599
I20260929 01:58:38.431700 8105 ts_manager.cc:194] Registered new tserver with Master: e1fcd1a5f6284d28b5f9f6eba530f0ba (127.0.71.65:38181)
I20260929 01:58:38.440999 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
WARNING: no policy specified for kudu/127.0.71.123@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.123@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.123 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.123 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.123@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.123@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.123 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.123 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:58:38.582194 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.123:44717
--webserver_interface=127.0.71.123
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.123
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:34213,127.0.71.85:33543,127.0.71.86:41399,127.0.71.87:42679,127.0.71.88:37589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:41357,127.0.71.125:39313,127.0.71.124:45665,127.0.71.123:44717 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:58:38.997982 8428 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:38.998456 8428 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:38.998618 8428 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:39.012789 8428 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:58:39.012965 8428 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:39.013054 8428 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:58:39.013136 8428 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:58:39.025890 8428 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:34213,127.0.71.85:33543,127.0.71.86:41399,127.0.71.87:42679,127.0.71.88:37589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:41357,127.0.71.125:39313,127.0.71.124:45665,127.0.71.123:44717
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.123
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.123:44717
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.0.71.123
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:39.027778 8428 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:39.029583 8428 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:39.040757 8433 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:39.041024 8436 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:39.040920 8434 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:39.042814 8428 server_base.cc:1061] running on GCE node
I20260929 01:58:39.044051 8428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:39.046371 8428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:39.049847 8428 hybrid_clock.cc:648] HybridClock initialized: now 1790647119049552 us; error 215 us; skew 500 ppm
I20260929 01:58:39.050427 8428 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:39 dist-test-slave-vz23 krb5kdc[7973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:39.054770 8428 init.cc:377] Logged in from keytab as kudu/127.0.71.123@KRBTEST.COM (short username kudu)
I20260929 01:58:39.057126 8428 webserver.cc:533] Webserver started at http://127.0.71.123:42985/ using document root <none> and password file <none>
I20260929 01:58:39.058055 8428 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:39.058235 8428 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:39.058679 8428 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:58:39.062017 8428 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-3/data/instance:
uuid: "069d9bc528f64d9db34469aeea35d348"
format_stamp: "Formatted at 2026-09-29 01:58:39 on dist-test-slave-vz23"
I20260929 01:58:39.063051 8428 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-3/wal/instance:
uuid: "069d9bc528f64d9db34469aeea35d348"
format_stamp: "Formatted at 2026-09-29 01:58:39 on dist-test-slave-vz23"
I20260929 01:58:39.070526 8428 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.008s sys 0.000s
I20260929 01:58:39.076393 8443 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:39.078552 8428 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.001s
I20260929 01:58:39.078871 8428 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "069d9bc528f64d9db34469aeea35d348"
format_stamp: "Formatted at 2026-09-29 01:58:39 on dist-test-slave-vz23"
I20260929 01:58:39.079144 8428 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:39.127663 8428 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:39.133754 8428 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:39.134130 8428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:39.159904 8428 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.123:44717
I20260929 01:58:39.159927 8494 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.123:44717 every 8 connection(s)
I20260929 01:58:39.161957 8428 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
I20260929 01:58:39.167548 8495 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:58:39.169351 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 8428
I20260929 01:58:39.177677 8495 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:58:39 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
Sep 29 01:58:39 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:58:39.200769 8495 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:58:39 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:58:39.217034 8495 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:58:39 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:58:39.237126 8495 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44717 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:58:39 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
I20260929 01:58:39.258351 8495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348: Bootstrap starting.
I20260929 01:58:39.262308 8495 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348: Neither blocks nor log segments found. Creating new log.
I20260929 01:58:39.263619 8495 log.cc:826] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348: Log is configured to *not* fsync() on all Append() calls
I20260929 01:58:39.267226 8495 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348: No bootstrap required, opened a new log
I20260929 01:58:39.273201 8495 raft_consensus.cc:359] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44717 } }
I20260929 01:58:39.273617 8495 raft_consensus.cc:385] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:58:39.273778 8495 raft_consensus.cc:740] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 069d9bc528f64d9db34469aeea35d348, State: Initialized, Role: FOLLOWER
I20260929 01:58:39.274622 8495 consensus_queue.cc:260] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44717 } }
I20260929 01:58:39.275806 8504 sys_catalog.cc:455] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44717 } } }
I20260929 01:58:39.276261 8504 sys_catalog.cc:458] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:39.277328 8495 sys_catalog.cc:565] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:58:39.289387 8515 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:58:39.289574 8515 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:58:39.298920 8428 master_runner.cc:429] Detected that this master 069d9bc528f64d9db34469aeea35d348 is joining an existing cluster
I20260929 01:58:39.299129 8428 catalog_manager.cc:1037] Shutting down the Catalog Manager
I20260929 01:58:39.300174 8428 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260929 01:58:39.300702 8428 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260929 01:58:39.300941 8428 tablet_replica.cc:333] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348: stopping tablet replica
I20260929 01:58:39.303354 8428 master_runner.cc:434] Initiating AddMaster RPC to add 127.0.71.123:44717
Sep 29 01:58:39 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647116, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
I20260929 01:58:39.380136 8182 catalog_manager.cc:7351] Initiating ChangeConfig request to add master 127.0.71.123:44717: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "069d9bc528f64d9db34469aeea35d348"
member_type: NON_VOTER
last_known_addr {
host: "127.0.71.123"
port: 44717
}
attrs {
promote: true
}
}
dest_uuid: "070f469f2361414ebb8f17e7ba40ddff"
cas_config_opid_index: -1
I20260929 01:58:39.381811 8182 consensus_queue.cc:237] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: true } }
I20260929 01:58:39.387576 8115 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a [term 1 FOLLOWER]: Refusing update from remote peer 070f469f2361414ebb8f17e7ba40ddff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 01:58:39.387750 8039 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [term 1 FOLLOWER]: Refusing update from remote peer 070f469f2361414ebb8f17e7ba40ddff: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20260929 01:58:39.387756 8168 consensus_peers.cc:597] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff -> Peer 069d9bc528f64d9db34469aeea35d348 (127.0.71.123:44717): Couldn't send request to peer 069d9bc528f64d9db34469aeea35d348. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260929 01:58:39.388378 8286 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:58:39.388859 8526 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [LEADER]: Connected to new peer: Peer: permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:58:39.395659 8526 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 069d9bc528f64d9db34469aeea35d348 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: true } } }
I20260929 01:58:39.396873 8039 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 069d9bc528f64d9db34469aeea35d348 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: true } } }
I20260929 01:58:39.397634 8115 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 069d9bc528f64d9db34469aeea35d348 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: true } } }
I20260929 01:58:39.401868 8526 catalog_manager.cc:7355] Successfully completed master ChangeConfig request to add master 127.0.71.123:44717
I20260929 01:58:39.403213 8428 master.cc:562] Master@127.0.71.123:44717 shutting down...
I20260929 01:58:39.420158 8530 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "070f469f2361414ebb8f17e7ba40ddff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: true } } }
I20260929 01:58:39.421566 8530 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:39.427773 8286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "070f469f2361414ebb8f17e7ba40ddff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: true } } }
I20260929 01:58:39.428140 8286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:39.431046 8286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "070f469f2361414ebb8f17e7ba40ddff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: true } } }
I20260929 01:58:39.431406 8286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:39.431823 8286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "070f469f2361414ebb8f17e7ba40ddff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: true } } }
I20260929 01:58:39.432092 8286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:39.433404 8531 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "070f469f2361414ebb8f17e7ba40ddff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: true } } }
I20260929 01:58:39.433786 8531 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:39.434136 8414 heartbeater.cc:499] Master 127.0.71.124:45665 was elected leader, sending a full tablet report...
I20260929 01:58:39.447644 8428 master.cc:584] Master@127.0.71.123:44717 shutdown complete.
I20260929 01:58:39.447779 8428 master_runner.cc:306] Clearing existing system tablet
I20260929 01:58:39.448210 8428 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:39.448341 8428 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:39.451156 8428 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260929 01:58:39.453152 8536 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:39.453581 8428 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:39.453896 8428 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.000s
I20260929 01:58:39.454010 8428 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "069d9bc528f64d9db34469aeea35d348"
format_stamp: "Formatted at 2026-09-29 01:58:39 on dist-test-slave-vz23"
I20260929 01:58:39.458076 8428 ts_tablet_manager.cc:1962] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:58:39.463842 8428 ts_tablet_manager.cc:1975] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 01:58:39.464130 8428 log.cc:1199] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260929 01:58:39.464715 8428 ts_tablet_manager.cc:1996] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348: Deleting consensus metadata
I20260929 01:58:39.464998 8428 master_runner.cc:316] Copying system tablet from 127.0.71.124:45665
I20260929 01:58:39.468130 8428 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348: tablet copy: Beginning tablet copy session from remote peer at address 127.0.71.124:45665
I20260929 01:58:39.484975 8202 tablet_copy_service.cc:140] P 070f469f2361414ebb8f17e7ba40ddff: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 069d9bc528f64d9db34469aeea35d348 ({username='kudu', principal='kudu/127.0.71.123@KRBTEST.COM'} at 127.0.0.1:41034)
I20260929 01:58:39.485320 8202 tablet_copy_service.cc:161] P 070f469f2361414ebb8f17e7ba40ddff: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 069d9bc528f64d9db34469aeea35d348 at {username='kudu', principal='kudu/127.0.71.123@KRBTEST.COM'} at 127.0.0.1:41034: session id = 069d9bc528f64d9db34469aeea35d348-00000000000000000000000000000000
I20260929 01:58:39.488641 8202 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 01:58:39.491201 8428 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 01:58:39.497665 8428 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348: tablet copy: Starting download of 0 data blocks...
I20260929 01:58:39.498077 8428 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348: tablet copy: Starting download of 1 WAL segments...
I20260929 01:58:39.501859 8428 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 01:58:39.507548 8202 tablet_copy_service.cc:342] P 070f469f2361414ebb8f17e7ba40ddff: Request end of tablet copy session 069d9bc528f64d9db34469aeea35d348-00000000000000000000000000000000 received from {username='kudu', principal='kudu/127.0.71.123@KRBTEST.COM'} at 127.0.0.1:41034
I20260929 01:58:39.508121 8202 tablet_copy_service.cc:434] P 070f469f2361414ebb8f17e7ba40ddff: ending tablet copy session 069d9bc528f64d9db34469aeea35d348-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 069d9bc528f64d9db34469aeea35d348
W20260929 01:58:39.517627 8428 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260929 01:58:39.521790 8428 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:39.528723 8543 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:39.529517 8544 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:39.538686 8428 server_base.cc:1061] running on GCE node
W20260929 01:58:39.539136 8546 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:39.539649 8428 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:39.540548 8428 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:39.541806 8428 hybrid_clock.cc:648] HybridClock initialized: now 1790647119541692 us; error 70 us; skew 500 ppm
I20260929 01:58:39.542099 8428 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:39 dist-test-slave-vz23 krb5kdc[7973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647119, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:39.544373 8428 init.cc:377] Logged in from keytab as kudu/127.0.71.123@KRBTEST.COM (short username kudu)
I20260929 01:58:39.545984 8428 webserver.cc:533] Webserver started at http://127.0.71.123:46719/ using document root <none> and password file <none>
I20260929 01:58:39.546360 8428 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:39.546486 8428 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:39.549660 8428 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:58:39.551684 8553 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:39.552206 8428 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:58:39.552376 8428 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "069d9bc528f64d9db34469aeea35d348"
format_stamp: "Formatted at 2026-09-29 01:58:39 on dist-test-slave-vz23"
I20260929 01:58:39.552608 8428 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:39.575467 8428 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:39.579838 8428 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:39.601101 8428 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.123:44717
I20260929 01:58:39.601125 8604 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.123:44717 every 8 connection(s)
I20260929 01:58:39.602607 8428 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
I20260929 01:58:39.607316 8605 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:58:39.610414 8605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348: Bootstrap starting.
I20260929 01:58:39.627552 8605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:58:39.628115 8605 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348: Bootstrap complete.
I20260929 01:58:39.629209 8605 raft_consensus.cc:359] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: true } }
I20260929 01:58:39.629491 8605 raft_consensus.cc:740] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 069d9bc528f64d9db34469aeea35d348, State: Initialized, Role: LEARNER
I20260929 01:58:39.629771 8605 consensus_queue.cc:260] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: true } }
I20260929 01:58:39.630776 8608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: true } } }
I20260929 01:58:39.631084 8608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348 [sys.catalog]: This master's current role is: LEARNER
I20260929 01:58:39.631129 8605 sys_catalog.cc:565] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:58:39.643268 8619 catalog_manager.cc:1308] Loaded cluster ID: 66a1b2eb2961442d98965e9214955dcc
I20260929 01:58:39.643426 8619 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348: loading cluster ID for follower catalog manager: success
I20260929 01:58:39.647207 8619 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348: acquiring CA information for follower catalog manager: success
I20260929 01:58:39.650095 8619 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:58:39.674606 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 8285
Sep 29 01:58:39 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647114, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 01:58:39.699980 8409 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:58:39.802503 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 8285
I20260929 01:58:39.827471 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:38181
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:41357,127.0.71.125:39313,127.0.71.124:45665
--webserver_port=46393
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:34213,127.0.71.85:33543,127.0.71.86:41399,127.0.71.87:42679,127.0.71.88:37589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:41357,127.0.71.125:39313,127.0.71.124:45665,127.0.71.123:44717 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:58:39.948933 8580 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260929 01:58:39.949914 8608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 070f469f2361414ebb8f17e7ba40ddff. Latest consensus state: current_term: 1 leader_uuid: "070f469f2361414ebb8f17e7ba40ddff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: true } } }
I20260929 01:58:39.950389 8608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348 [sys.catalog]: This master's current role is: LEARNER
W20260929 01:58:40.252658 8623 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:40.253018 8623 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:40.253130 8623 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:40.262672 8623 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:40.262897 8623 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:58:40.276052 8623 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:34213,127.0.71.85:33543,127.0.71.86:41399,127.0.71.87:42679,127.0.71.88:37589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:38181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=46393
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:41357,127.0.71.125:39313,127.0.71.124:45665,127.0.71.123:44717
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:40.277994 8623 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:40.279627 8623 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:40.292972 8630 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:40.292972 8629 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:40.293754 8632 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:40.295071 8623 server_base.cc:1061] running on GCE node
I20260929 01:58:40.295925 8623 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:40.297487 8623 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:40.300953 8623 hybrid_clock.cc:648] HybridClock initialized: now 1790647120300701 us; error 179 us; skew 500 ppm
I20260929 01:58:40.301553 8623 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:40 dist-test-slave-vz23 krb5kdc[7973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:40.305965 8623 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:58:40.308521 8623 webserver.cc:533] Webserver started at http://127.0.71.65:46393/ using document root <none> and password file <none>
I20260929 01:58:40.309515 8623 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:40.309695 8623 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:40.317407 8623 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 01:58:40.321626 8639 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:40.323882 8623 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:58:40.324244 8623 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "e1fcd1a5f6284d28b5f9f6eba530f0ba"
format_stamp: "Formatted at 2026-09-29 01:58:38 on dist-test-slave-vz23"
I20260929 01:58:40.325268 8623 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:40.345467 8623 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:40.352360 8623 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:40.352833 8623 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:40.354187 8623 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:58:40.356388 8623 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:58:40.356609 8623 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.001s sys 0.000s
I20260929 01:58:40.356786 8623 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:58:40.356890 8623 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:40.364777 8532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "070f469f2361414ebb8f17e7ba40ddff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: true } } }
I20260929 01:58:40.365206 8532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:40.367352 8656 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff: attempting to promote NON_VOTER 069d9bc528f64d9db34469aeea35d348 to VOTER
I20260929 01:58:40.369100 8656 consensus_queue.cc:237] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: false } }
I20260929 01:58:40.372469 8115 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a [term 1 FOLLOWER]: Refusing update from remote peer 070f469f2361414ebb8f17e7ba40ddff: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:58:40.372568 8580 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348 [term 1 LEARNER]: Refusing update from remote peer 070f469f2361414ebb8f17e7ba40ddff: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:58:40.373569 8653 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [LEADER]: Connected to new peer: Peer: permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:58:40.375011 8532 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [LEADER]: Connected to new peer: Peer: permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:58:40.375566 8039 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [term 1 FOLLOWER]: Refusing update from remote peer 070f469f2361414ebb8f17e7ba40ddff: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:58:40.376359 8532 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [LEADER]: Connected to new peer: Peer: permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:58:40.381016 8653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "070f469f2361414ebb8f17e7ba40ddff" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: true } } } pending_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: false } } }
I20260929 01:58:40.381531 8653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:40.382257 8653 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 069d9bc528f64d9db34469aeea35d348 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: false } } }
I20260929 01:58:40.382547 8580 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 069d9bc528f64d9db34469aeea35d348 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: false } } }
I20260929 01:58:40.385433 8039 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 069d9bc528f64d9db34469aeea35d348 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: false } } }
I20260929 01:58:40.388926 8115 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 069d9bc528f64d9db34469aeea35d348 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: false } } }
I20260929 01:58:40.389461 8608 sys_catalog.cc:455] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "070f469f2361414ebb8f17e7ba40ddff" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: false } } }
I20260929 01:58:40.389834 8608 sys_catalog.cc:458] T 00000000000000000000000000000000 P 069d9bc528f64d9db34469aeea35d348 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:40.389701 8532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "070f469f2361414ebb8f17e7ba40ddff" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: false } } }
I20260929 01:58:40.389994 8532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:40.390352 8532 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "070f469f2361414ebb8f17e7ba40ddff" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: false } } }
I20260929 01:58:40.390563 8532 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:40.391117 8656 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "070f469f2361414ebb8f17e7ba40ddff" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: false } } }
I20260929 01:58:40.391321 8656 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:40.394512 8699 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "070f469f2361414ebb8f17e7ba40ddff" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: false } } }
I20260929 01:58:40.395078 8699 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:40.396140 8700 sys_catalog.cc:455] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "070f469f2361414ebb8f17e7ba40ddff" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: false } } }
I20260929 01:58:40.396763 8700 sys_catalog.cc:458] T 00000000000000000000000000000000 P 24103e5cfa7e4e0d9788bfe029d7a44a [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:40.418004 8623 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:38181
I20260929 01:58:40.418088 8758 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:38181 every 8 connection(s)
I20260929 01:58:40.420008 8623 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:58:40.426705 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 8623
Sep 29 01:58:40 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
Sep 29 01:58:40 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:58:40 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:58:40 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647120, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:58:40.474622 8761 heartbeater.cc:344] Connected to a master server at 127.0.71.123:44717
I20260929 01:58:40.475072 8760 heartbeater.cc:344] Connected to a master server at 127.0.71.126:41357
I20260929 01:58:40.475236 8761 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:40.475288 8760 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:40.477228 8760 heartbeater.cc:507] Master 127.0.71.126:41357 requested a full tablet report, sending...
I20260929 01:58:40.477262 8761 heartbeater.cc:507] Master 127.0.71.123:44717 requested a full tablet report, sending...
I20260929 01:58:40.480424 8028 ts_manager.cc:194] Re-registered known tserver with Master: e1fcd1a5f6284d28b5f9f6eba530f0ba (127.0.71.65:38181)
I20260929 01:58:40.480794 8570 ts_manager.cc:194] Registered new tserver with Master: e1fcd1a5f6284d28b5f9f6eba530f0ba (127.0.71.65:38181)
I20260929 01:58:40.481262 8759 heartbeater.cc:344] Connected to a master server at 127.0.71.124:45665
I20260929 01:58:40.481467 8759 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:40.482295 8759 heartbeater.cc:507] Master 127.0.71.124:45665 requested a full tablet report, sending...
I20260929 01:58:40.484099 8182 ts_manager.cc:194] Re-registered known tserver with Master: e1fcd1a5f6284d28b5f9f6eba530f0ba (127.0.71.65:38181)
I20260929 01:58:40.485479 8182 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:53505
I20260929 01:58:40.485860 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:58:40.487187 8762 heartbeater.cc:344] Connected to a master server at 127.0.71.125:39313
I20260929 01:58:40.487411 8762 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:40.488339 8762 heartbeater.cc:507] Master 127.0.71.125:39313 requested a full tablet report, sending...
I20260929 01:58:40.490357 8105 ts_manager.cc:194] Re-registered known tserver with Master: e1fcd1a5f6284d28b5f9f6eba530f0ba (127.0.71.65:38181)
I20260929 01:58:40.501034 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:58:40.501292 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 7997
I20260929 01:58:40.502621 8059 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:58:40.628638 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 7997
I20260929 01:58:40.655948 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:41357
--webserver_interface=127.0.71.126
--webserver_port=39613
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:34213,127.0.71.85:33543,127.0.71.86:41399,127.0.71.87:42679,127.0.71.88:37589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:41357,127.0.71.125:39313,127.0.71.124:45665
--master_addresses=127.0.71.126:41357,127.0.71.125:39313,127.0.71.124:45665,127.0.71.123:44717 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:58:40.770185 8166 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.126:41357: connect: Connection refused (error 111)
W20260929 01:58:40.772102 8166 consensus_peers.cc:597] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff -> Peer 5cc926876156403ba1430e293b6c57f5 (127.0.71.126:41357): Couldn't send request to peer 5cc926876156403ba1430e293b6c57f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.71.126:41357: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260929 01:58:41.084549 8772 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:41.084971 8772 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:41.085079 8772 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:41.095625 8772 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:58:41.095849 8772 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:41.095964 8772 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:58:41.096143 8772 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:58:41.111747 8772 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:34213,127.0.71.85:33543,127.0.71.86:41399,127.0.71.87:42679,127.0.71.88:37589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:41357,127.0.71.125:39313,127.0.71.124:45665,127.0.71.123:44717
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:41357
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=39613
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:41.113903 8772 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:41.115604 8772 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:41.127326 8780 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:41.127439 8782 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:41.127369 8779 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:41.128867 8772 server_base.cc:1061] running on GCE node
I20260929 01:58:41.129820 8772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:41.132239 8772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:41.135689 8772 hybrid_clock.cc:648] HybridClock initialized: now 1790647121135531 us; error 120 us; skew 500 ppm
I20260929 01:58:41.136101 8772 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:41 dist-test-slave-vz23 krb5kdc[7973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:41.140033 8772 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:58:41.142616 8772 webserver.cc:533] Webserver started at http://127.0.71.126:39613/ using document root <none> and password file <none>
I20260929 01:58:41.143622 8772 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:41.143839 8772 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:41.153666 8772 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.001s
I20260929 01:58:41.157577 8789 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:41.159653 8772 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:58:41.159960 8772 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "5cc926876156403ba1430e293b6c57f5"
format_stamp: "Formatted at 2026-09-29 01:58:35 on dist-test-slave-vz23"
I20260929 01:58:41.161291 8772 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:41.181280 8772 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:41.193820 8772 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:41.194211 8772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:41.220235 8772 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:41357
I20260929 01:58:41.220225 8840 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:41357 every 8 connection(s)
I20260929 01:58:41.222504 8772 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:58:41.231923 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 8772
I20260929 01:58:41.232311 8841 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:58:41.232332 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 8623
I20260929 01:58:41.238179 8841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5: Bootstrap starting.
I20260929 01:58:41.255036 8754 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:58:41.257628 8653 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "070f469f2361414ebb8f17e7ba40ddff" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: false } } }
I20260929 01:58:41.257997 8653 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: This master's current role is: LEADER
I20260929 01:58:41.267867 8841 log.cc:826] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5: Log is configured to *not* fsync() on all Append() calls
I20260929 01:58:41.286351 8841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:58:41.287238 8841 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5: Bootstrap complete.
I20260929 01:58:41.297989 8841 raft_consensus.cc:359] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: false } }
I20260929 01:58:41.299221 8841 raft_consensus.cc:740] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 5cc926876156403ba1430e293b6c57f5, State: Initialized, Role: FOLLOWER
I20260929 01:58:41.300218 8841 consensus_queue.cc:260] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: false } }
I20260929 01:58:41.301503 8851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: false } } }
I20260929 01:58:41.302014 8851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:41.303090 8841 sys_catalog.cc:565] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:58:41.304243 8851 sys_catalog.cc:455] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 070f469f2361414ebb8f17e7ba40ddff. Latest consensus state: current_term: 1 leader_uuid: "070f469f2361414ebb8f17e7ba40ddff" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: false } } }
I20260929 01:58:41.304729 8851 sys_catalog.cc:458] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:58:41.318073 8862 catalog_manager.cc:1308] Loaded cluster ID: 66a1b2eb2961442d98965e9214955dcc
I20260929 01:58:41.318329 8862 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5: loading cluster ID for follower catalog manager: success
I20260929 01:58:41.324954 8862 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5: acquiring CA information for follower catalog manager: success
I20260929 01:58:41.328617 8862 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 5cc926876156403ba1430e293b6c57f5: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:58:41.398756 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 8623
I20260929 01:58:41.417829 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:38181
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:41357,127.0.71.125:39313,127.0.71.124:45665
--webserver_port=46393
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:34213,127.0.71.85:33543,127.0.71.86:41399,127.0.71.87:42679,127.0.71.88:37589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:41357,127.0.71.125:39313,127.0.71.124:45665,127.0.71.123:44717 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:58:41.701875 8846 sys_catalog.cc:455] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "070f469f2361414ebb8f17e7ba40ddff" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "5cc926876156403ba1430e293b6c57f5" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41357 } } peers { permanent_uuid: "24103e5cfa7e4e0d9788bfe029d7a44a" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 39313 } } peers { permanent_uuid: "070f469f2361414ebb8f17e7ba40ddff" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 45665 } } peers { permanent_uuid: "069d9bc528f64d9db34469aeea35d348" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 44717 } attrs { promote: false } } }
I20260929 01:58:41.702273 8846 sys_catalog.cc:458] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff [sys.catalog]: This master's current role is: LEADER
W20260929 01:58:41.851712 8868 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:41.852223 8868 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:41.852425 8868 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:41.862818 8868 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:41.863166 8868 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:58:41.877218 8868 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:34213,127.0.71.85:33543,127.0.71.86:41399,127.0.71.87:42679,127.0.71.88:37589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:38181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=46393
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:41357,127.0.71.125:39313,127.0.71.124:45665,127.0.71.123:44717
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:41.879290 8868 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:41.881072 8868 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:41.896790 8873 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:41.897064 8876 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:41.896799 8874 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:41.898002 8868 server_base.cc:1061] running on GCE node
I20260929 01:58:41.898828 8868 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:41.900512 8868 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:41.904093 8868 hybrid_clock.cc:648] HybridClock initialized: now 1790647121903824 us; error 158 us; skew 500 ppm
I20260929 01:58:41.904817 8868 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:41 dist-test-slave-vz23 krb5kdc[7973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:41.909404 8868 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:58:41.912149 8868 webserver.cc:533] Webserver started at http://127.0.71.65:46393/ using document root <none> and password file <none>
I20260929 01:58:41.913182 8868 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:41.913321 8868 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:41.920545 8868 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:58:41.924939 8883 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:41.926946 8868 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.004s
I20260929 01:58:41.927317 8868 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "e1fcd1a5f6284d28b5f9f6eba530f0ba"
format_stamp: "Formatted at 2026-09-29 01:58:38 on dist-test-slave-vz23"
I20260929 01:58:41.928228 8868 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:41.956815 8868 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:41.962332 8868 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:41.962714 8868 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:41.964054 8868 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:58:41.966048 8868 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:58:41.966163 8868 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:41.966243 8868 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:58:41.966300 8868 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:42.012959 8868 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:38181
I20260929 01:58:42.013031 8995 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:38181 every 8 connection(s)
I20260929 01:58:42.014885 8868 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:58:42.025483 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 8868
Sep 29 01:58:42 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
Sep 29 01:58:42 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:58:42 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:58:42 dist-test-slave-vz23 krb5kdc[7973](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647121, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:58:42.073987 8996 heartbeater.cc:344] Connected to a master server at 127.0.71.124:45665
I20260929 01:58:42.074627 8996 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:42.076027 8996 heartbeater.cc:507] Master 127.0.71.124:45665 requested a full tablet report, sending...
I20260929 01:58:42.077042 8998 heartbeater.cc:344] Connected to a master server at 127.0.71.123:44717
I20260929 01:58:42.077276 8998 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:42.077962 8998 heartbeater.cc:507] Master 127.0.71.123:44717 requested a full tablet report, sending...
I20260929 01:58:42.079125 8997 heartbeater.cc:344] Connected to a master server at 127.0.71.126:41357
I20260929 01:58:42.079311 8997 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:42.080032 8997 heartbeater.cc:507] Master 127.0.71.126:41357 requested a full tablet report, sending...
I20260929 01:58:42.080271 8570 ts_manager.cc:194] Re-registered known tserver with Master: e1fcd1a5f6284d28b5f9f6eba530f0ba (127.0.71.65:38181)
I20260929 01:58:42.082366 8182 ts_manager.cc:194] Re-registered known tserver with Master: e1fcd1a5f6284d28b5f9f6eba530f0ba (127.0.71.65:38181)
I20260929 01:58:42.082719 8806 ts_manager.cc:194] Registered new tserver with Master: e1fcd1a5f6284d28b5f9f6eba530f0ba (127.0.71.65:38181)
I20260929 01:58:42.084123 8182 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:45745
I20260929 01:58:42.088792 8999 heartbeater.cc:344] Connected to a master server at 127.0.71.125:39313
I20260929 01:58:42.089022 8999 heartbeater.cc:461] Registering TS with master...
I20260929 01:58:42.089694 8999 heartbeater.cc:507] Master 127.0.71.125:39313 requested a full tablet report, sending...
I20260929 01:58:42.091791 8105 ts_manager.cc:194] Re-registered known tserver with Master: e1fcd1a5f6284d28b5f9f6eba530f0ba (127.0.71.65:38181)
I20260929 01:58:42.124847 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 8868
W20260929 01:58:42.126340 9014 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:58:42.121957 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:40078 (local address 127.0.71.65:38181)
0929 01:58:42.122805 (+ 848us) server_negotiation.cc:207] Beginning negotiation
0929 01:58:42.122815 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0929 01:58:42.122850 (+ 35us) server_negotiation.cc:408] Connection header received
0929 01:58:42.122933 (+ 83us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:58:42.122941 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:58:42.123244 (+ 303us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:58:42.123456 (+ 212us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:58:42.124778 (+ 1322us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40078: BlockingRecv error: recv got EOF from 127.0.0.1:40078 (error 108)
Metrics: {"server-negotiator.queue_time_us":641,"thread_start_us":484,"threads_started":1}
W20260929 01:58:42.128150 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 8868: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:38181: server requires authentication, but client does not have Kerberos credentials available
W20260929 01:58:42.128293 9014 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:58:42.125888 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:40084 (local address 127.0.71.65:38181)
0929 01:58:42.126685 (+ 797us) server_negotiation.cc:207] Beginning negotiation
0929 01:58:42.126695 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0929 01:58:42.126721 (+ 26us) server_negotiation.cc:408] Connection header received
0929 01:58:42.126828 (+ 107us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:58:42.126835 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:58:42.126929 (+ 94us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:58:42.127050 (+ 121us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:58:42.128040 (+ 990us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:40084: BlockingRecv error: recv got EOF from 127.0.0.1:40084 (error 108)
Metrics: {"server-negotiator.queue_time_us":695,"thread_start_us":643,"threads_started":1}
I20260929 01:58:42.128410 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 8868
I20260929 01:58:42.145324 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:38181
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:41357,127.0.71.125:39313,127.0.71.124:45665
--webserver_port=46393
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:34213,127.0.71.85:33543,127.0.71.86:41399,127.0.71.87:42679,127.0.71.88:37589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:41357,127.0.71.125:39313,127.0.71.124:45665,127.0.71.123:44717
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:58:42.564270 9016 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:42.564774 9016 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:42.564960 9016 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:58:42.565033 9016 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:42.574620 9016 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:42.574965 9016 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:58:42.587098 9016 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:34213,127.0.71.85:33543,127.0.71.86:41399,127.0.71.87:42679,127.0.71.88:37589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:38181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=46393
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:41357,127.0.71.125:39313,127.0.71.124:45665,127.0.71.123:44717
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:42.589169 9016 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:42.590975 9016 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:42.603914 9021 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:42.604830 9022 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:42.610790 9024 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:42.611176 9016 server_base.cc:1061] running on GCE node
I20260929 01:58:42.611909 9016 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:42.613435 9016 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:42.616925 9016 hybrid_clock.cc:648] HybridClock initialized: now 1790647122616634 us; error 166 us; skew 500 ppm
I20260929 01:58:42.617458 9016 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:42 dist-test-slave-vz23 krb5kdc[7973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647122, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:42.621717 9016 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:58:42.623941 9016 webserver.cc:533] Webserver started at http://127.0.71.65:46393/ using document root <none> and password file <none>
I20260929 01:58:42.625002 9016 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:42.625116 9016 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:42.632696 9016 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 01:58:42.636857 9031 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:42.638736 9016 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.000s sys 0.003s
I20260929 01:58:42.638993 9016 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "e1fcd1a5f6284d28b5f9f6eba530f0ba"
format_stamp: "Formatted at 2026-09-29 01:58:38 on dist-test-slave-vz23"
I20260929 01:58:42.639914 9016 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:42.658591 9016 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:42.664405 9016 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:42.664927 9016 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:42.666293 9016 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:58:42.668872 9016 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:58:42.669029 9016 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:42.669201 9016 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:58:42.669307 9016 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:58:42.669927 9016 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:58:42.670037 9016 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:58:42.670133 9016 fault_injection.cc:59] injecting fault for kudu (pid 9016): FLAGS_fault_before_start (process will exit)
I20260929 01:58:42.687491 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:38181
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:41357,127.0.71.125:39313,127.0.71.124:45665
--webserver_port=46393
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:34213,127.0.71.85:33543,127.0.71.86:41399,127.0.71.87:42679,127.0.71.88:37589
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:41357,127.0.71.125:39313,127.0.71.124:45665,127.0.71.123:44717 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:58:43.094842 9039 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:58:43.095307 9039 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:58:43.095474 9039 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:58:43.105146 9039 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:58:43.105456 9039 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:58:43.117743 9039 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:34213,127.0.71.85:33543,127.0.71.86:41399,127.0.71.87:42679,127.0.71.88:37589
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:38181
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=46393
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:41357,127.0.71.125:39313,127.0.71.124:45665,127.0.71.123:44717
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:58:43.119851 9039 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:58:43.121623 9039 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:58:43.135525 9047 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:43.135668 9045 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:58:43.135540 9044 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:58:43.136531 9039 server_base.cc:1061] running on GCE node
I20260929 01:58:43.137257 9039 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:58:43.138703 9039 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:58:43.142184 9039 hybrid_clock.cc:648] HybridClock initialized: now 1790647123141986 us; error 121 us; skew 500 ppm
I20260929 01:58:43.142721 9039 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:58:43 dist-test-slave-vz23 krb5kdc[7973](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647123, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:58:43.147076 9039 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:58:43.149318 9039 webserver.cc:533] Webserver started at http://127.0.71.65:46393/ using document root <none> and password file <none>
I20260929 01:58:43.150235 9039 fs_manager.cc:362] Metadata directory not provided
I20260929 01:58:43.150403 9039 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:58:43.157743 9039 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.003s sys 0.000s
I20260929 01:58:43.161722 9054 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:43.163708 9039 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.004s sys 0.000s
I20260929 01:58:43.164041 9039 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "e1fcd1a5f6284d28b5f9f6eba530f0ba"
format_stamp: "Formatted at 2026-09-29 01:58:38 on dist-test-slave-vz23"
I20260929 01:58:43.165062 9039 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:58:43.196519 9039 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:58:43.201855 9039 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:58:43.202214 9039 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:58:43.203496 9039 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:58:43.205621 9039 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:58:43.205789 9039 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:43.205945 9039 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:58:43.206041 9039 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:58:43.249003 9039 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:38181
I20260929 01:58:43.249068 9166 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:38181 every 8 connection(s)
I20260929 01:58:43.251196 9039 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:58:43.252914 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 9039
I20260929 01:58:43.253309 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 9039
W20260929 01:58:43.291441 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 9039: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:38181: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:58:43.291637 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 9039
I20260929 01:58:43.311978 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 8772
W20260929 01:58:43.312964 9181 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:58:43.281770 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:45509 (local address 127.0.71.123:44717)
0929 01:58:43.295522 (+ 13752us) server_negotiation.cc:207] Beginning negotiation
0929 01:58:43.295531 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0929 01:58:43.311718 (+ 16187us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:45509: BlockingRecv error: recv got EOF from 127.0.71.65:45509 (error 108)
Metrics: {"server-negotiator.queue_time_us":13573,"thread_start_us":13394,"threads_started":1}
W20260929 01:58:43.312964 9171 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:58:43.255905 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:34923 (local address 127.0.71.126:41357)
0929 01:58:43.266834 (+ 10929us) server_negotiation.cc:207] Beginning negotiation
0929 01:58:43.266846 (+ 12us) server_negotiation.cc:400] Waiting for connection header
0929 01:58:43.271946 (+ 5100us) server_negotiation.cc:408] Connection header received
0929 01:58:43.277054 (+ 5108us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:58:43.277069 (+ 15us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:58:43.277248 (+ 179us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:58:43.277387 (+ 139us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:58:43.290958 (+ 13571us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:58:43.296279 (+ 5321us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:58:43.311729 (+ 15450us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:34923: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":10651,"thread_start_us":10488,"threads_started":1}
W20260929 01:58:43.312980 9174 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:58:43.257937 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:41037 (local address 127.0.71.124:45665)
0929 01:58:43.278582 (+ 20645us) server_negotiation.cc:207] Beginning negotiation
0929 01:58:43.278592 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0929 01:58:43.287126 (+ 8534us) server_negotiation.cc:408] Connection header received
0929 01:58:43.287280 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:58:43.287293 (+ 13us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:58:43.287435 (+ 142us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:58:43.287644 (+ 209us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:58:43.291447 (+ 3803us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:58:43.294459 (+ 3012us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:58:43.311736 (+ 17277us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:41037: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":20391,"thread_start_us":20212,"threads_started":1}
W20260929 01:58:43.314656 9179 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:58:43.279934 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:36479 (local address 127.0.71.125:39313)
0929 01:58:43.294989 (+ 15055us) server_negotiation.cc:207] Beginning negotiation
0929 01:58:43.294998 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0929 01:58:43.313707 (+ 18709us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:36479: BlockingRecv error: recv got EOF from 127.0.71.65:36479 (error 108)
Metrics: {"server-negotiator.queue_time_us":14892,"thread_start_us":15624,"threads_started":1}
W20260929 01:58:43.315464 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 8772: Not authorized: Client connection negotiation failed: client connection to 127.0.71.126:41357: server requires authentication, but client does not have Kerberos credentials available
W20260929 01:58:43.315582 9171 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:58:43.313831 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:33548 (local address 127.0.71.126:41357)
0929 01:58:43.314088 (+ 257us) server_negotiation.cc:207] Beginning negotiation
0929 01:58:43.314093 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0929 01:58:43.314115 (+ 22us) server_negotiation.cc:408] Connection header received
0929 01:58:43.314186 (+ 71us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:58:43.314191 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:58:43.314267 (+ 76us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:58:43.314420 (+ 153us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:58:43.315337 (+ 917us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:33548: BlockingRecv error: recv got EOF from 127.0.0.1:33548 (error 108)
Metrics: {"server-negotiator.queue_time_us":90}
I20260929 01:58:43.315645 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 8772
I20260929 01:58:43.340817 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 8073
I20260929 01:58:43.342280 8135 generic_service.cc:196] Checking for leaks (request via RPC)
W20260929 01:58:43.416031 8166 consensus_peers.cc:597] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff -> Peer 5cc926876156403ba1430e293b6c57f5 (127.0.71.126:41357): Couldn't send request to peer 5cc926876156403ba1430e293b6c57f5. Status: Network error: Client connection negotiation failed: client connection to 127.0.71.126:41357: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 01:58:43.472689 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 8073
W20260929 01:58:43.496785 8169 connection.cc:573] client connection to 127.0.71.125:39313 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:58:43.497483 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 8150
W20260929 01:58:43.498659 8169 consensus_peers.cc:597] T 00000000000000000000000000000000 P 070f469f2361414ebb8f17e7ba40ddff -> Peer 24103e5cfa7e4e0d9788bfe029d7a44a (127.0.71.125:39313): Couldn't send request to peer 24103e5cfa7e4e0d9788bfe029d7a44a. Status: Network error: Client connection negotiation failed: client connection to 127.0.71.125:39313: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 01:58:43.498888 8212 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:58:43.631878 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 8150
W20260929 01:58:43.658179 8556 connection.cc:573] server connection from 127.0.0.1:40352 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:58:43.658670 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 8428
I20260929 01:58:43.660153 8600 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:58:43.789899 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 8428
2026-09-29T01:58:43Z chronyd exiting
2026-09-29T01:58:43Z chronyd exiting
2026-09-29T01:58:43Z chronyd exiting
2026-09-29T01:58:43Z chronyd exiting
2026-09-29T01:58:43Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestAddMaster/8 (11686 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/9
Loading random data
Initializing database '/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:58:43 dist-test-slave-vz23 krb5kdc[9189](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:58:43 dist-test-slave-vz23 krb5kdc[9189](info): set up 2 sockets
Sep 29 01:58:43 dist-test-slave-vz23 krb5kdc[9189](info): commencing operation
krb5kdc: starting...
W20260929 01:58:46.073932 285 mini_kdc.cc:121] Time spent starting KDC: real 2.191s user 0.004s sys 0.036s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 01:58:46 dist-test-slave-vz23 krb5kdc[9189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
WARNING: no policy specified for hive/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "hive/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/hive_127.0.0.1.keytab.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/hive_127.0.0.1.keytab.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/krb5.conf
2026-09-29 01:58:55: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
Sep 29 01:58:58 dist-test-slave-vz23 krb5kdc[9189](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790647138, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 01:59:06 dist-test-slave-vz23 krb5kdc[9189](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790647146, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 01:59:06 dist-test-slave-vz23 krb5kdc[9189](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790647146, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:59:07.667610 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:33791
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:33791,127.0.71.125:34895,127.0.71.124:43671 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:59:08.087749 9486 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:59:08.088150 9486 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:59:08.088369 9486 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:59:08.098935 9486 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:59:08.099102 9486 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:59:08.099174 9486 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:59:08.099220 9486 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:59:08.113332 9486 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:33791,127.0.71.125:34895,127.0.71.124:43671
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:33791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:59:08.116087 9486 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:59:08.118997 9486 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:59:08.132947 9492 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:08.133111 9494 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:08.132812 9491 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:59:08.134403 9486 server_base.cc:1061] running on GCE node
I20260929 01:59:08.136191 9486 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:59:08.136535 9486 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:59:08.136718 9486 hybrid_clock.cc:648] HybridClock initialized: now 1790647148136647 us; error 0 us; skew 500 ppm
I20260929 01:59:08.137210 9486 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:59:08 dist-test-slave-vz23 krb5kdc[9189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:59:08.148737 9486 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:59:08.151605 9486 webserver.cc:533] Webserver started at http://127.0.71.126:39359/ using document root <none> and password file <none>
I20260929 01:59:08.152529 9486 fs_manager.cc:362] Metadata directory not provided
I20260929 01:59:08.152668 9486 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:59:08.153020 9486 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:59:08.156142 9486 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "e647e46c807f4ad8abb25157f59fb74d"
format_stamp: "Formatted at 2026-09-29 01:59:08 on dist-test-slave-vz23"
I20260929 01:59:08.157367 9486 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "e647e46c807f4ad8abb25157f59fb74d"
format_stamp: "Formatted at 2026-09-29 01:59:08 on dist-test-slave-vz23"
I20260929 01:59:08.165180 9486 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.000s
I20260929 01:59:08.169994 9500 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:08.172485 9486 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:59:08.172796 9486 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "e647e46c807f4ad8abb25157f59fb74d"
format_stamp: "Formatted at 2026-09-29 01:59:08 on dist-test-slave-vz23"
I20260929 01:59:08.173049 9486 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:59:08.224687 9486 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:59:08.233608 9486 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:59:08.234627 9486 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:59:08.269936 9486 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:33791
I20260929 01:59:08.269963 9551 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:33791 every 8 connection(s)
I20260929 01:59:08.272550 9486 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:59:08.278077 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 9486
I20260929 01:59:08.278398 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:59:08.281844 9552 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:59:08.294124 9552 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } has no permanent_uuid. Determining permanent_uuid...
WARNING: no policy specified for kudu/127.0.71.125@KRBTEST.COM; defaulting to no policy
Sep 29 01:59:08 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.125@KRBTEST.COM" created.
I20260929 01:59:08.331372 9552 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:59:08.333668 9501 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.125:34895: connect: Connection refused (error 111)
W20260929 01:59:08.336373 9552 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:34895: Network error: Client connection negotiation failed: client connection to 127.0.71.125:34895: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:59:08.386231 9552 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } attempt: 1
W20260929 01:59:08.389298 9552 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:34895: Network error: Client connection negotiation failed: client connection to 127.0.71.125:34895: connect: Connection refused (error 111)
WARNING: no policy specified for HTTP/127.0.71.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:59:08.436936 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:34895
--webserver_interface=127.0.71.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:33791,127.0.71.125:34895,127.0.71.124:43671 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:59:08.457648 9552 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } attempt: 2
W20260929 01:59:08.460909 9552 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:34895: Network error: Client connection negotiation failed: client connection to 127.0.71.125:34895: connect: Connection refused (error 111)
I20260929 01:59:08.552278 9552 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } attempt: 3
W20260929 01:59:08.555085 9552 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:34895: Network error: Client connection negotiation failed: client connection to 127.0.71.125:34895: connect: Connection refused (error 111)
I20260929 01:59:08.698464 9552 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } attempt: 4
W20260929 01:59:08.701547 9552 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:34895: Network error: Client connection negotiation failed: client connection to 127.0.71.125:34895: connect: Connection refused (error 111)
W20260929 01:59:08.840267 9561 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:59:08.840758 9561 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:59:08.840956 9561 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:59:08.851167 9561 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:59:08.851308 9561 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:59:08.851413 9561 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:59:08.851517 9561 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:59:08.863723 9561 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:33791,127.0.71.125:34895,127.0.71.124:43671
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:34895
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:59:08.865880 9561 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:59:08.867729 9561 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:59:08.880915 9566 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:08.880908 9567 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:08.881004 9569 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:59:08.882591 9561 server_base.cc:1061] running on GCE node
I20260929 01:59:08.883695 9561 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:59:08.883894 9561 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:59:08.884063 9561 hybrid_clock.cc:648] HybridClock initialized: now 1790647148884017 us; error 0 us; skew 500 ppm
I20260929 01:59:08.884552 9561 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:59:08 dist-test-slave-vz23 krb5kdc[9189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:59:08.889348 9561 init.cc:377] Logged in from keytab as kudu/127.0.71.125@KRBTEST.COM (short username kudu)
I20260929 01:59:08.892755 9561 webserver.cc:533] Webserver started at http://127.0.71.125:38323/ using document root <none> and password file <none>
I20260929 01:59:08.893846 9561 fs_manager.cc:362] Metadata directory not provided
I20260929 01:59:08.894068 9561 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:59:08.894593 9561 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:59:08.897728 9561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-1/data/instance:
uuid: "8f7c23b6f007491f904a323c402755a1"
format_stamp: "Formatted at 2026-09-29 01:59:08 on dist-test-slave-vz23"
I20260929 01:59:08.898813 9561 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-1/wal/instance:
uuid: "8f7c23b6f007491f904a323c402755a1"
format_stamp: "Formatted at 2026-09-29 01:59:08 on dist-test-slave-vz23"
I20260929 01:59:08.906605 9561 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.005s
I20260929 01:59:08.911422 9575 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:08.913265 9561 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:59:08.913549 9561 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "8f7c23b6f007491f904a323c402755a1"
format_stamp: "Formatted at 2026-09-29 01:59:08 on dist-test-slave-vz23"
I20260929 01:59:08.913834 9561 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:59:08.935164 9561 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:59:08.939949 9561 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:59:08.940315 9561 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:59:08.965227 9626 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:34895 every 8 connection(s)
I20260929 01:59:08.965227 9561 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:34895
I20260929 01:59:08.967270 9561 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:59:08.972858 9627 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:59:08.972985 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 9561
I20260929 01:59:08.973168 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:59:08.983647 9627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:59:08 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:59:09.000874 9552 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } attempt: 5
WARNING: no policy specified for kudu/127.0.71.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.124@KRBTEST.COM" created.
I20260929 01:59:09.019107 9627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:59:09 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:59:09 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:59:09.037503 9552 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:59:09.040799 9552 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43671: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43671: connect: Connection refused (error 111)
I20260929 01:59:09.047547 9627 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:59:09.049427 9578 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43671: connect: Connection refused (error 111)
W20260929 01:59:09.051899 9627 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43671: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43671: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.124@KRBTEST.COM" created.
I20260929 01:59:09.092120 9552 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } attempt: 1
W20260929 01:59:09.095012 9552 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43671: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43671: connect: Connection refused (error 111)
I20260929 01:59:09.101359 9627 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } attempt: 1
W20260929 01:59:09.104240 9627 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43671: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43671: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:59:09.134534 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:43671
--webserver_interface=127.0.71.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:33791,127.0.71.125:34895,127.0.71.124:43671 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:59:09.163375 9552 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } attempt: 2
W20260929 01:59:09.166182 9552 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43671: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43671: connect: Connection refused (error 111)
I20260929 01:59:09.172591 9627 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } attempt: 2
W20260929 01:59:09.175053 9627 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43671: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43671: connect: Connection refused (error 111)
I20260929 01:59:09.266395 9627 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } attempt: 3
W20260929 01:59:09.269367 9627 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43671: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43671: connect: Connection refused (error 111)
I20260929 01:59:09.272541 9552 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } attempt: 3
W20260929 01:59:09.275054 9552 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43671: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43671: connect: Connection refused (error 111)
I20260929 01:59:09.412683 9627 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } attempt: 4
W20260929 01:59:09.415588 9627 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43671: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43671: connect: Connection refused (error 111)
I20260929 01:59:09.452387 9552 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } attempt: 4
W20260929 01:59:09.455217 9552 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:43671: Network error: Client connection negotiation failed: client connection to 127.0.71.124:43671: connect: Connection refused (error 111)
W20260929 01:59:09.542356 9638 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:59:09.542865 9638 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:59:09.543061 9638 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:59:09.552708 9638 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:59:09.552909 9638 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:59:09.553043 9638 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:59:09.553134 9638 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:59:09.565644 9638 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:33791,127.0.71.125:34895,127.0.71.124:43671
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:43671
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:59:09.567723 9638 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:59:09.569536 9638 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:59:09.581736 9644 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:09.582121 9646 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:09.581657 9643 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:59:09.583540 9638 server_base.cc:1061] running on GCE node
I20260929 01:59:09.584730 9638 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:59:09.584924 9638 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:59:09.585008 9638 hybrid_clock.cc:648] HybridClock initialized: now 1790647149584987 us; error 0 us; skew 500 ppm
I20260929 01:59:09.585350 9638 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:59:09 dist-test-slave-vz23 krb5kdc[9189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:59:09.589533 9638 init.cc:377] Logged in from keytab as kudu/127.0.71.124@KRBTEST.COM (short username kudu)
I20260929 01:59:09.592329 9638 webserver.cc:533] Webserver started at http://127.0.71.124:42077/ using document root <none> and password file <none>
I20260929 01:59:09.593338 9638 fs_manager.cc:362] Metadata directory not provided
I20260929 01:59:09.593479 9638 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:59:09.593825 9638 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:59:09.596879 9638 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-2/data/instance:
uuid: "ae6a09befe714482a4fa52a23f43cf5d"
format_stamp: "Formatted at 2026-09-29 01:59:09 on dist-test-slave-vz23"
I20260929 01:59:09.597837 9638 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-2/wal/instance:
uuid: "ae6a09befe714482a4fa52a23f43cf5d"
format_stamp: "Formatted at 2026-09-29 01:59:09 on dist-test-slave-vz23"
I20260929 01:59:09.605466 9638 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.003s sys 0.004s
I20260929 01:59:09.610590 9652 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:09.612932 9638 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.007s sys 0.000s
I20260929 01:59:09.613220 9638 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "ae6a09befe714482a4fa52a23f43cf5d"
format_stamp: "Formatted at 2026-09-29 01:59:09 on dist-test-slave-vz23"
I20260929 01:59:09.613507 9638 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:59:09.655980 9638 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:59:09.662045 9638 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:59:09.662462 9638 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:59:09.689728 9638 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:43671
I20260929 01:59:09.689749 9703 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:43671 every 8 connection(s)
I20260929 01:59:09.691305 9638 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:59:09.696566 9704 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:59:09.701659 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 9638
I20260929 01:59:09.702085 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:59:09.703909 9704 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:59:09.714905 9627 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } attempt: 5
Sep 29 01:59:09 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:59:09.732959 9552 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } attempt: 5
Sep 29 01:59:09 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:59:09 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:59:09 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:59:09 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:59:09.756142 9704 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:59:09.762157 9627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1: Bootstrap starting.
I20260929 01:59:09.766928 9552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d: Bootstrap starting.
I20260929 01:59:09.768014 9627 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1: Neither blocks nor log segments found. Creating new log.
I20260929 01:59:09.770084 9627 log.cc:826] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1: Log is configured to *not* fsync() on all Append() calls
I20260929 01:59:09.773490 9552 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d: Neither blocks nor log segments found. Creating new log.
Sep 29 01:59:09 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:59:09.776119 9552 log.cc:826] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d: Log is configured to *not* fsync() on all Append() calls
Sep 29 01:59:09 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:59:09.780423 9552 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d: No bootstrap required, opened a new log
I20260929 01:59:09.780417 9627 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1: No bootstrap required, opened a new log
I20260929 01:59:09.789101 9704 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } has no permanent_uuid. Determining permanent_uuid...
I20260929 01:59:09.789884 9552 raft_consensus.cc:359] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } }
I20260929 01:59:09.790298 9552 raft_consensus.cc:385] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:59:09.790479 9552 raft_consensus.cc:740] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e647e46c807f4ad8abb25157f59fb74d, State: Initialized, Role: FOLLOWER
I20260929 01:59:09.791388 9552 consensus_queue.cc:260] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } }
I20260929 01:59:09.791406 9627 raft_consensus.cc:359] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } }
I20260929 01:59:09.791908 9627 raft_consensus.cc:385] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:59:09.792096 9627 raft_consensus.cc:740] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8f7c23b6f007491f904a323c402755a1, State: Initialized, Role: FOLLOWER
I20260929 01:59:09.793160 9627 consensus_queue.cc:260] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } }
I20260929 01:59:09.797243 9717 sys_catalog.cc:455] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } }
I20260929 01:59:09.798504 9718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } }
I20260929 01:59:09.799050 9718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:09.799152 9717 sys_catalog.cc:458] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:09.798962 9552 sys_catalog.cc:565] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:59:09.800262 9627 sys_catalog.cc:565] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [sys.catalog]: configured and running, proceeding with master startup.
Sep 29 01:59:09 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647149, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:59:09.823263 9704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d: Bootstrap starting.
I20260929 01:59:09.830128 9704 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d: Neither blocks nor log segments found. Creating new log.
I20260929 01:59:09.832631 9704 log.cc:826] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:59:09.821723 9486 master_runner.cc:187] Error getting master registration for 127.0.71.125:34895: OK, instance_id { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" instance_seqno: 1790647148958494 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:59:09.837100 9486 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:59:09.841323 9704 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d: No bootstrap required, opened a new log
W20260929 01:59:09.842887 9732 catalog_manager.cc:1607] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:59:09.843173 9732 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:59:09.853358 9704 raft_consensus.cc:359] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } }
I20260929 01:59:09.854063 9704 raft_consensus.cc:385] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:59:09.854290 9704 raft_consensus.cc:740] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: ae6a09befe714482a4fa52a23f43cf5d, State: Initialized, Role: FOLLOWER
I20260929 01:59:09.855414 9704 consensus_queue.cc:260] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } }
I20260929 01:59:09.863497 9561 master_runner.cc:187] Error getting master registration for 127.0.71.124:43671: OK, instance_id { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" instance_seqno: 1790647149683301 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:59:09.871151 9561 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:59:09.874615 9742 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:59:09.874821 9742 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:59:09.875584 9744 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } }
I20260929 01:59:09.875978 9744 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:09.877454 9704 sys_catalog.cc:565] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:59:09.902693 9638 master_runner.cc:264] No leader master found from master ae6a09befe714482a4fa52a23f43cf5d
I20260929 01:59:09.902881 9638 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:59:09.907066 9755 catalog_manager.cc:1607] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:59:09.907361 9755 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 01:59:09.915201 9658 tablet.cc:2433] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:59:09.943871 9581 tablet.cc:2433] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:59:09.992141 9506 tablet.cc:2433] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:59:10.005810 9717 raft_consensus.cc:493] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:59:10.006186 9717 raft_consensus.cc:515] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } }
I20260929 01:59:10.008481 9717 leader_election.cc:290] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8f7c23b6f007491f904a323c402755a1 (127.0.71.125:34895), ae6a09befe714482a4fa52a23f43cf5d (127.0.71.124:43671)
I20260929 01:59:10.009949 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e647e46c807f4ad8abb25157f59fb74d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f7c23b6f007491f904a323c402755a1" is_pre_election: true
I20260929 01:59:10.010599 9602 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e647e46c807f4ad8abb25157f59fb74d in term 0.
I20260929 01:59:10.013139 9679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e647e46c807f4ad8abb25157f59fb74d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae6a09befe714482a4fa52a23f43cf5d" is_pre_election: true
I20260929 01:59:10.014014 9679 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate e647e46c807f4ad8abb25157f59fb74d in term 0.
I20260929 01:59:10.011734 9501 leader_election.cc:304] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f7c23b6f007491f904a323c402755a1, e647e46c807f4ad8abb25157f59fb74d; no voters:
I20260929 01:59:10.016188 9717 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:59:10.016544 9717 raft_consensus.cc:493] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:59:10.016763 9717 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:59:10.024020 9717 raft_consensus.cc:515] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } }
I20260929 01:59:10.028659 9602 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e647e46c807f4ad8abb25157f59fb74d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8f7c23b6f007491f904a323c402755a1"
I20260929 01:59:10.029016 9602 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:59:10.033583 9602 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e647e46c807f4ad8abb25157f59fb74d in term 1.
I20260929 01:59:10.034726 9501 leader_election.cc:304] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8f7c23b6f007491f904a323c402755a1, e647e46c807f4ad8abb25157f59fb74d; no voters:
I20260929 01:59:10.040055 9679 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "e647e46c807f4ad8abb25157f59fb74d" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "ae6a09befe714482a4fa52a23f43cf5d"
I20260929 01:59:10.040377 9679 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:59:10.045141 9679 raft_consensus.cc:2468] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate e647e46c807f4ad8abb25157f59fb74d in term 1.
I20260929 01:59:10.046777 9717 leader_election.cc:290] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [CANDIDATE]: Term 1 election: Requested vote from peers 8f7c23b6f007491f904a323c402755a1 (127.0.71.125:34895), ae6a09befe714482a4fa52a23f43cf5d (127.0.71.124:43671)
I20260929 01:59:10.053257 9717 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:59:10.053997 9717 raft_consensus.cc:697] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [term 1 LEADER]: Becoming Leader. State: Replica: e647e46c807f4ad8abb25157f59fb74d, State: Running, Role: LEADER
I20260929 01:59:10.054952 9717 consensus_queue.cc:237] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } }
I20260929 01:59:10.071635 9757 sys_catalog.cc:455] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e647e46c807f4ad8abb25157f59fb74d. Latest consensus state: current_term: 1 leader_uuid: "e647e46c807f4ad8abb25157f59fb74d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } }
I20260929 01:59:10.072214 9757 sys_catalog.cc:458] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [sys.catalog]: This master's current role is: LEADER
I20260929 01:59:10.087531 9759 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:59:10.091078 9759 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:59:10.115494 9679 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [term 1 FOLLOWER]: Refusing update from remote peer e647e46c807f4ad8abb25157f59fb74d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:59:10.117161 9757 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:59:10.126174 9602 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [term 1 FOLLOWER]: Refusing update from remote peer e647e46c807f4ad8abb25157f59fb74d: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:59:10.131935 9757 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:59:10.165266 9744 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader e647e46c807f4ad8abb25157f59fb74d. Latest consensus state: current_term: 1 leader_uuid: "e647e46c807f4ad8abb25157f59fb74d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } }
I20260929 01:59:10.165678 9744 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:10.173369 9760 mvcc.cc:204] Tried to move back new op lower bound from 7334490726857060352 to 7334490726643601408. Current Snapshot: MvccSnapshot[applied={T|T < 7334490726857060352}]
I20260929 01:59:10.178009 9761 mvcc.cc:204] Tried to move back new op lower bound from 7334490726857060352 to 7334490726643601408. Current Snapshot: MvccSnapshot[applied={T|T < 7334490726857060352}]
I20260929 01:59:10.186034 9718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e647e46c807f4ad8abb25157f59fb74d. Latest consensus state: current_term: 1 leader_uuid: "e647e46c807f4ad8abb25157f59fb74d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } }
I20260929 01:59:10.186494 9718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:10.190691 9717 sys_catalog.cc:455] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e647e46c807f4ad8abb25157f59fb74d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } }
I20260929 01:59:10.191051 9717 sys_catalog.cc:458] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [sys.catalog]: This master's current role is: LEADER
I20260929 01:59:10.199666 9759 catalog_manager.cc:1396] Generated new cluster ID: ed6b7a0757884b82802fb0d43c0d88c5
I20260929 01:59:10.199842 9759 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:59:10.200832 9744 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e647e46c807f4ad8abb25157f59fb74d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } }
I20260929 01:59:10.201148 9744 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:10.216037 9762 mvcc.cc:204] Tried to move back new op lower bound from 7334490726857060352 to 7334490726643601408. Current Snapshot: MvccSnapshot[applied={T|T < 7334490726857060352}]
I20260929 01:59:10.216958 9757 sys_catalog.cc:455] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e647e46c807f4ad8abb25157f59fb74d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } }
I20260929 01:59:10.217455 9757 sys_catalog.cc:458] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [sys.catalog]: This master's current role is: LEADER
I20260929 01:59:10.246243 9718 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e647e46c807f4ad8abb25157f59fb74d" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } }
I20260929 01:59:10.246572 9718 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:10.277179 9759 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 01:59:10.279874 9759 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:59:10.315201 9759 catalog_manager.cc:6219] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d: Generated new TSK 0
I20260929 01:59:10.317173 9759 catalog_manager.cc:1563] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:59:10.538285 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:33791,127.0.71.125:34895,127.0.71.124:43671
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:59:10.877564 9742 catalog_manager.cc:1308] Loaded cluster ID: ed6b7a0757884b82802fb0d43c0d88c5
I20260929 01:59:10.877727 9742 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1: loading cluster ID for follower catalog manager: success
I20260929 01:59:10.881650 9742 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1: acquiring CA information for follower catalog manager: success
I20260929 01:59:10.884794 9742 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:59:10.909679 9755 catalog_manager.cc:1308] Loaded cluster ID: ed6b7a0757884b82802fb0d43c0d88c5
I20260929 01:59:10.909850 9755 catalog_manager.cc:1601] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d: loading cluster ID for follower catalog manager: success
I20260929 01:59:10.912858 9755 catalog_manager.cc:1623] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d: acquiring CA information for follower catalog manager: success
I20260929 01:59:10.915313 9755 catalog_manager.cc:1651] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 01:59:11.000917 9773 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:59:11.001293 9773 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:59:11.001540 9773 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:59:11.014777 9773 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:59:11.015019 9773 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:59:11.027384 9773 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:33791,127.0.71.125:34895,127.0.71.124:43671
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:59:11.029290 9773 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:59:11.030952 9773 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:59:11.043668 9778 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:11.043766 9779 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:11.044818 9781 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:59:11.045614 9773 server_base.cc:1061] running on GCE node
I20260929 01:59:11.046643 9773 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:59:11.046936 9773 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:59:11.047083 9773 hybrid_clock.cc:648] HybridClock initialized: now 1790647151047049 us; error 0 us; skew 500 ppm
I20260929 01:59:11.047484 9773 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:59:11 dist-test-slave-vz23 krb5kdc[9189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:59:11.052662 9773 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:59:11.055217 9773 webserver.cc:533] Webserver started at http://127.0.71.65:36029/ using document root <none> and password file <none>
I20260929 01:59:11.056267 9773 fs_manager.cc:362] Metadata directory not provided
I20260929 01:59:11.056732 9773 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:59:11.057483 9773 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:59:11.061332 9773 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "25146d82715b417f826b236363302518"
format_stamp: "Formatted at 2026-09-29 01:59:11 on dist-test-slave-vz23"
I20260929 01:59:11.062408 9773 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "25146d82715b417f826b236363302518"
format_stamp: "Formatted at 2026-09-29 01:59:11 on dist-test-slave-vz23"
I20260929 01:59:11.069707 9773 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.006s sys 0.001s
I20260929 01:59:11.074888 9787 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:11.076887 9773 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.002s
I20260929 01:59:11.077270 9773 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "25146d82715b417f826b236363302518"
format_stamp: "Formatted at 2026-09-29 01:59:11 on dist-test-slave-vz23"
I20260929 01:59:11.077533 9773 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:59:11.105098 9773 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:59:11.110631 9773 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:59:11.111035 9773 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:59:11.112376 9773 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:59:11.114598 9773 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:59:11.114742 9773 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:11.114938 9773 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:59:11.115034 9773 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:11.160691 9773 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:37775
I20260929 01:59:11.160750 9899 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:37775 every 8 connection(s)
I20260929 01:59:11.162470 9773 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:59:11.169162 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 9773
I20260929 01:59:11.169446 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
Sep 29 01:59:11 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:59:11 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:59:11 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:59:11.232080 9900 heartbeater.cc:344] Connected to a master server at 127.0.71.124:43671
I20260929 01:59:11.232303 9901 heartbeater.cc:344] Connected to a master server at 127.0.71.126:33791
I20260929 01:59:11.232666 9900 heartbeater.cc:461] Registering TS with master...
I20260929 01:59:11.232667 9901 heartbeater.cc:461] Registering TS with master...
I20260929 01:59:11.232688 9902 heartbeater.cc:344] Connected to a master server at 127.0.71.125:34895
I20260929 01:59:11.232934 9902 heartbeater.cc:461] Registering TS with master...
I20260929 01:59:11.233954 9900 heartbeater.cc:507] Master 127.0.71.124:43671 requested a full tablet report, sending...
I20260929 01:59:11.233991 9902 heartbeater.cc:507] Master 127.0.71.125:34895 requested a full tablet report, sending...
I20260929 01:59:11.234036 9901 heartbeater.cc:507] Master 127.0.71.126:33791 requested a full tablet report, sending...
I20260929 01:59:11.237252 9517 ts_manager.cc:194] Registered new tserver with Master: 25146d82715b417f826b236363302518 (127.0.71.65:37775)
I20260929 01:59:11.237306 9592 ts_manager.cc:194] Registered new tserver with Master: 25146d82715b417f826b236363302518 (127.0.71.65:37775)
I20260929 01:59:11.237306 9668 ts_manager.cc:194] Registered new tserver with Master: 25146d82715b417f826b236363302518 (127.0.71.65:37775)
I20260929 01:59:11.239552 9517 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:47843
I20260929 01:59:11.246871 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
WARNING: no policy specified for kudu/127.0.71.123@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.123@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.123 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.123 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.123@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.123@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.123 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.123 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:59:11.410387 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.123:33165
--webserver_interface=127.0.71.123
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.123
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:33791,127.0.71.125:34895,127.0.71.124:43671,127.0.71.123:33165 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:59:11.821740 9915 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:59:11.822211 9915 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:59:11.822386 9915 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:59:11.831614 9915 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:59:11.831828 9915 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:59:11.831976 9915 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:59:11.832077 9915 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:59:11.845376 9915 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:33791,127.0.71.125:34895,127.0.71.124:43671,127.0.71.123:33165
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.123
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.123:33165
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.0.71.123
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:59:11.847828 9915 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:59:11.849792 9915 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:59:11.861358 9920 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:11.861495 9923 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:11.862287 9921 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:59:11.863591 9915 server_base.cc:1061] running on GCE node
I20260929 01:59:11.865338 9915 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:59:11.865607 9915 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:59:11.865702 9915 hybrid_clock.cc:648] HybridClock initialized: now 1790647151865669 us; error 0 us; skew 500 ppm
I20260929 01:59:11.866189 9915 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:59:11 dist-test-slave-vz23 krb5kdc[9189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:59:11.872108 9915 init.cc:377] Logged in from keytab as kudu/127.0.71.123@KRBTEST.COM (short username kudu)
I20260929 01:59:11.875490 9915 webserver.cc:533] Webserver started at http://127.0.71.123:40209/ using document root <none> and password file <none>
I20260929 01:59:11.876648 9915 fs_manager.cc:362] Metadata directory not provided
I20260929 01:59:11.876824 9915 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:59:11.877350 9915 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:59:11.880898 9915 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-3/data/instance:
uuid: "e33df7eb69674f6ba64c48539cd9aa84"
format_stamp: "Formatted at 2026-09-29 01:59:11 on dist-test-slave-vz23"
I20260929 01:59:11.882095 9915 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-3/wal/instance:
uuid: "e33df7eb69674f6ba64c48539cd9aa84"
format_stamp: "Formatted at 2026-09-29 01:59:11 on dist-test-slave-vz23"
I20260929 01:59:11.890272 9915 fs_manager.cc:696] Time spent creating directory manager: real 0.007s user 0.007s sys 0.001s
I20260929 01:59:11.896584 9929 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:11.898567 9915 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:59:11.898794 9915 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "e33df7eb69674f6ba64c48539cd9aa84"
format_stamp: "Formatted at 2026-09-29 01:59:11 on dist-test-slave-vz23"
I20260929 01:59:11.899021 9915 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:59:11.926802 9915 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:59:11.932082 9915 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:59:11.932374 9915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:59:11.957440 9915 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.123:33165
I20260929 01:59:11.957455 9981 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.123:33165 every 8 connection(s)
I20260929 01:59:11.959396 9915 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
I20260929 01:59:11.965520 9982 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:59:11.966267 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 9915
I20260929 01:59:11.977348 9982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:59:11 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
Sep 29 01:59:11 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:59:12.006703 9982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:59:12 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:59:12.024714 9982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:59:12 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:59:12.045428 9982 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33165 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:59:12 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647151, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
I20260929 01:59:12.073009 9982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84: Bootstrap starting.
I20260929 01:59:12.077287 9982 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84: Neither blocks nor log segments found. Creating new log.
I20260929 01:59:12.078951 9982 log.cc:826] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84: Log is configured to *not* fsync() on all Append() calls
I20260929 01:59:12.083776 9982 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84: No bootstrap required, opened a new log
I20260929 01:59:12.090297 9982 raft_consensus.cc:359] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33165 } }
I20260929 01:59:12.091392 9982 raft_consensus.cc:385] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:59:12.091634 9982 raft_consensus.cc:740] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e33df7eb69674f6ba64c48539cd9aa84, State: Initialized, Role: FOLLOWER
I20260929 01:59:12.092712 9982 consensus_queue.cc:260] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33165 } }
I20260929 01:59:12.093991 9991 sys_catalog.cc:455] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33165 } } }
I20260929 01:59:12.094345 9991 sys_catalog.cc:458] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:12.095206 9982 sys_catalog.cc:565] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:59:12.107812 10002 catalog_manager.cc:1607] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:59:12.108042 10002 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:59:12.108338 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 9773
I20260929 01:59:12.121062 9915 master_runner.cc:429] Detected that this master e33df7eb69674f6ba64c48539cd9aa84 is joining an existing cluster
I20260929 01:59:12.121236 9915 catalog_manager.cc:1037] Shutting down the Catalog Manager
Sep 29 01:59:12 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 01:59:12.123957 9915 raft_consensus.cc:2243] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260929 01:59:12.124531 9915 raft_consensus.cc:2272] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260929 01:59:12.124766 9915 tablet_replica.cc:333] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84: stopping tablet replica
I20260929 01:59:12.127583 9915 master_runner.cc:434] Initiating AddMaster RPC to add 127.0.71.123:33165
I20260929 01:59:12.133176 9895 generic_service.cc:196] Checking for leaks (request via RPC)
Sep 29 01:59:12 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647148, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
I20260929 01:59:12.224530 9517 catalog_manager.cc:7351] Initiating ChangeConfig request to add master 127.0.71.123:33165: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84"
member_type: NON_VOTER
last_known_addr {
host: "127.0.71.123"
port: 33165
}
attrs {
promote: true
}
}
dest_uuid: "e647e46c807f4ad8abb25157f59fb74d"
cas_config_opid_index: -1
I20260929 01:59:12.227224 9517 consensus_queue.cc:237] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } }
W20260929 01:59:12.233000 9502 consensus_peers.cc:597] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d -> Peer e33df7eb69674f6ba64c48539cd9aa84 (127.0.71.123:33165): Couldn't send request to peer e33df7eb69674f6ba64c48539cd9aa84. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260929 01:59:12.234272 9601 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [term 1 FOLLOWER]: Refusing update from remote peer e647e46c807f4ad8abb25157f59fb74d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 01:59:12.234653 9678 raft_consensus.cc:1275] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [term 1 FOLLOWER]: Refusing update from remote peer e647e46c807f4ad8abb25157f59fb74d: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 01:59:12.235867 9934 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [LEADER]: Connected to new peer: Peer: permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:59:12.236289 10016 consensus_queue.cc:1048] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:59:12.252266 9934 raft_consensus.cc:2955] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e33df7eb69674f6ba64c48539cd9aa84 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } } }
I20260929 01:59:12.253105 9678 raft_consensus.cc:2955] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e33df7eb69674f6ba64c48539cd9aa84 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } } }
I20260929 01:59:12.258759 9934 catalog_manager.cc:7355] Successfully completed master ChangeConfig request to add master 127.0.71.123:33165
I20260929 01:59:12.260196 9915 master.cc:562] Master@127.0.71.123:33165 shutting down...
I20260929 01:59:12.279076 10016 sys_catalog.cc:455] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "e647e46c807f4ad8abb25157f59fb74d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } } }
I20260929 01:59:12.279727 10016 sys_catalog.cc:458] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [sys.catalog]: This master's current role is: LEADER
I20260929 01:59:12.281700 10019 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e647e46c807f4ad8abb25157f59fb74d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } } }
I20260929 01:59:12.282040 10019 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:12.281526 10020 sys_catalog.cc:455] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e647e46c807f4ad8abb25157f59fb74d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } } }
I20260929 01:59:12.286823 10020 sys_catalog.cc:458] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [sys.catalog]: This master's current role is: LEADER
I20260929 01:59:12.287266 9915 master.cc:584] Master@127.0.71.123:33165 shutdown complete.
I20260929 01:59:12.287393 9915 master_runner.cc:306] Clearing existing system tablet
I20260929 01:59:12.288051 9915 fs_manager.cc:362] Metadata directory not provided
I20260929 01:59:12.288208 9915 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:59:12.295598 9901 heartbeater.cc:499] Master 127.0.71.126:33791 was elected leader, sending a full tablet report...
I20260929 01:59:12.299606 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 9773
I20260929 01:59:12.303176 9934 sys_catalog.cc:455] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e647e46c807f4ad8abb25157f59fb74d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } } }
I20260929 01:59:12.303624 9934 sys_catalog.cc:458] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [sys.catalog]: This master's current role is: LEADER
I20260929 01:59:12.303858 9601 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER e33df7eb69674f6ba64c48539cd9aa84 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } } }
I20260929 01:59:12.308218 9915 fs_manager.cc:714] Time spent opening directory manager: real 0.019s user 0.003s sys 0.000s
I20260929 01:59:12.315850 10025 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "e647e46c807f4ad8abb25157f59fb74d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } } }
I20260929 01:59:12.316686 10025 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:12.320245 10026 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:12.320825 9915 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:59:12.321182 9915 fs_manager.cc:730] Time spent opening block manager: real 0.012s user 0.003s sys 0.007s
I20260929 01:59:12.321326 9915 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "e33df7eb69674f6ba64c48539cd9aa84"
format_stamp: "Formatted at 2026-09-29 01:59:11 on dist-test-slave-vz23"
I20260929 01:59:12.325227 9915 ts_tablet_manager.cc:1962] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 01:59:12.327713 9501 connection.cc:573] server connection from 127.0.71.65:47843 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:59:12.328820 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:37775
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:33791,127.0.71.125:34895,127.0.71.124:43671
--webserver_port=36029
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--tserver_master_addrs=127.0.71.126:33791,127.0.71.125:34895,127.0.71.124:43671,127.0.71.123:33165 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:59:12.334100 9915 ts_tablet_manager.cc:1975] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 01:59:12.334347 9915 log.cc:1199] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260929 01:59:12.334872 9915 ts_tablet_manager.cc:1996] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84: Deleting consensus metadata
I20260929 01:59:12.335186 9915 master_runner.cc:316] Copying system tablet from 127.0.71.126:33791
I20260929 01:59:12.338539 9915 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84: tablet copy: Beginning tablet copy session from remote peer at address 127.0.71.126:33791
I20260929 01:59:12.363686 9537 tablet_copy_service.cc:140] P e647e46c807f4ad8abb25157f59fb74d: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer e33df7eb69674f6ba64c48539cd9aa84 ({username='kudu', principal='kudu/127.0.71.123@KRBTEST.COM'} at 127.0.0.1:34902)
I20260929 01:59:12.364097 9537 tablet_copy_service.cc:161] P e647e46c807f4ad8abb25157f59fb74d: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer e33df7eb69674f6ba64c48539cd9aa84 at {username='kudu', principal='kudu/127.0.71.123@KRBTEST.COM'} at 127.0.0.1:34902: session id = e33df7eb69674f6ba64c48539cd9aa84-00000000000000000000000000000000
I20260929 01:59:12.369501 9537 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 01:59:12.372831 9915 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 01:59:12.383708 9915 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84: tablet copy: Starting download of 0 data blocks...
I20260929 01:59:12.384235 9915 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84: tablet copy: Starting download of 1 WAL segments...
I20260929 01:59:12.388296 9915 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 01:59:12.394321 9537 tablet_copy_service.cc:342] P e647e46c807f4ad8abb25157f59fb74d: Request end of tablet copy session e33df7eb69674f6ba64c48539cd9aa84-00000000000000000000000000000000 received from {username='kudu', principal='kudu/127.0.71.123@KRBTEST.COM'} at 127.0.0.1:34902
I20260929 01:59:12.394616 9537 tablet_copy_service.cc:434] P e647e46c807f4ad8abb25157f59fb74d: ending tablet copy session e33df7eb69674f6ba64c48539cd9aa84-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer e33df7eb69674f6ba64c48539cd9aa84
I20260929 01:59:12.409862 9915 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:59:12.413749 10034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:12.415776 10035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:59:12.433558 9915 server_base.cc:1061] running on GCE node
W20260929 01:59:12.433725 10037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:59:12.434231 9915 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:59:12.434331 9915 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:59:12.434414 9915 hybrid_clock.cc:648] HybridClock initialized: now 1790647152434368 us; error 0 us; skew 500 ppm
I20260929 01:59:12.434664 9915 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:59:12 dist-test-slave-vz23 krb5kdc[9189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:59:12.437659 9915 init.cc:377] Logged in from keytab as kudu/127.0.71.123@KRBTEST.COM (short username kudu)
I20260929 01:59:12.449751 9915 webserver.cc:533] Webserver started at http://127.0.71.123:35355/ using document root <none> and password file <none>
I20260929 01:59:12.450393 9915 fs_manager.cc:362] Metadata directory not provided
I20260929 01:59:12.450623 9915 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:59:12.454845 9915 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:59:12.457878 10043 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:12.458803 9915 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.003s sys 0.000s
I20260929 01:59:12.459142 9915 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "e33df7eb69674f6ba64c48539cd9aa84"
format_stamp: "Formatted at 2026-09-29 01:59:11 on dist-test-slave-vz23"
I20260929 01:59:12.459450 9915 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:59:12.501731 9915 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:59:12.507264 9915 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:59:12.562644 10094 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.123:33165 every 8 connection(s)
I20260929 01:59:12.562647 9915 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.123:33165
I20260929 01:59:12.564154 9915 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
I20260929 01:59:12.572057 10095 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:59:12.575218 10095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84: Bootstrap starting.
I20260929 01:59:12.594244 10095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:59:12.594987 10095 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84: Bootstrap complete.
I20260929 01:59:12.596278 10095 raft_consensus.cc:359] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } }
I20260929 01:59:12.596697 10095 raft_consensus.cc:740] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: e33df7eb69674f6ba64c48539cd9aa84, State: Initialized, Role: LEARNER
I20260929 01:59:12.597062 10095 consensus_queue.cc:260] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } }
I20260929 01:59:12.598210 10098 sys_catalog.cc:455] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } } }
I20260929 01:59:12.598620 10095 sys_catalog.cc:565] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:59:12.598650 10098 sys_catalog.cc:458] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84 [sys.catalog]: This master's current role is: LEARNER
I20260929 01:59:12.612957 10109 catalog_manager.cc:1308] Loaded cluster ID: ed6b7a0757884b82802fb0d43c0d88c5
I20260929 01:59:12.613124 10109 catalog_manager.cc:1601] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84: loading cluster ID for follower catalog manager: success
I20260929 01:59:12.616626 10109 catalog_manager.cc:1623] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84: acquiring CA information for follower catalog manager: success
I20260929 01:59:12.619663 10109 catalog_manager.cc:1651] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:59:12.651484 10070 raft_consensus.cc:1217] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260929 01:59:12.652534 10098 sys_catalog.cc:455] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader e647e46c807f4ad8abb25157f59fb74d. Latest consensus state: current_term: 1 leader_uuid: "e647e46c807f4ad8abb25157f59fb74d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } } }
I20260929 01:59:12.653481 10098 sys_catalog.cc:458] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84 [sys.catalog]: This master's current role is: LEARNER
W20260929 01:59:12.887434 10033 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:59:12.887825 10033 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:59:12.887931 10033 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:59:12.898285 10033 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:59:12.898531 10033 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:59:12.912345 10033 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:37775
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=36029
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:33791,127.0.71.125:34895,127.0.71.124:43671,127.0.71.123:33165
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:59:12.914346 10033 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:59:12.915935 10033 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:59:12.929576 10116 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:12.929759 10115 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:12.929553 10118 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:59:12.930552 10033 server_base.cc:1061] running on GCE node
I20260929 01:59:12.931329 10033 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:59:12.931558 10033 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:59:12.931684 10033 hybrid_clock.cc:648] HybridClock initialized: now 1790647152931636 us; error 0 us; skew 500 ppm
I20260929 01:59:12.932082 10033 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:59:12 dist-test-slave-vz23 krb5kdc[9189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:59:12.936602 10033 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:59:12.938993 10033 webserver.cc:533] Webserver started at http://127.0.71.65:36029/ using document root <none> and password file <none>
I20260929 01:59:12.939940 10033 fs_manager.cc:362] Metadata directory not provided
I20260929 01:59:12.940131 10033 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:59:12.948374 10033 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 01:59:12.953354 10124 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:12.955585 10033 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.001s
I20260929 01:59:12.955878 10033 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "25146d82715b417f826b236363302518"
format_stamp: "Formatted at 2026-09-29 01:59:11 on dist-test-slave-vz23"
I20260929 01:59:12.957077 10033 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:59:12.976761 10033 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:59:12.982270 10033 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:59:12.982617 10033 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:59:12.983886 10033 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:59:12.986124 10033 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:59:12.986251 10033 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:12.986351 10033 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:59:12.986462 10033 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:13.038115 10033 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:37775
I20260929 01:59:13.038190 10236 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:37775 every 8 connection(s)
I20260929 01:59:13.040133 10033 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:59:13.047096 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 10033
Sep 29 01:59:13 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:59:13 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
Sep 29 01:59:13 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:59:13.121120 10237 heartbeater.cc:344] Connected to a master server at 127.0.71.123:33165
I20260929 01:59:13.121726 10237 heartbeater.cc:461] Registering TS with master...
I20260929 01:59:13.123849 10239 heartbeater.cc:344] Connected to a master server at 127.0.71.126:33791
I20260929 01:59:13.124104 10239 heartbeater.cc:461] Registering TS with master...
I20260929 01:59:13.125161 10239 heartbeater.cc:507] Master 127.0.71.126:33791 requested a full tablet report, sending...
I20260929 01:59:13.127760 10237 heartbeater.cc:507] Master 127.0.71.123:33165 requested a full tablet report, sending...
I20260929 01:59:13.127919 9517 ts_manager.cc:194] Re-registered known tserver with Master: 25146d82715b417f826b236363302518 (127.0.71.65:37775)
I20260929 01:59:13.129191 9517 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:45409
Sep 29 01:59:13 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647152, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:59:13.131490 10060 ts_manager.cc:194] Registered new tserver with Master: 25146d82715b417f826b236363302518 (127.0.71.65:37775)
I20260929 01:59:13.131906 10240 heartbeater.cc:344] Connected to a master server at 127.0.71.125:34895
I20260929 01:59:13.132112 10240 heartbeater.cc:461] Registering TS with master...
I20260929 01:59:13.132547 10240 heartbeater.cc:507] Master 127.0.71.125:34895 requested a full tablet report, sending...
I20260929 01:59:13.134760 9592 ts_manager.cc:194] Re-registered known tserver with Master: 25146d82715b417f826b236363302518 (127.0.71.65:37775)
I20260929 01:59:13.138599 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:59:13.141487 10238 heartbeater.cc:344] Connected to a master server at 127.0.71.124:43671
I20260929 01:59:13.141716 10238 heartbeater.cc:461] Registering TS with master...
I20260929 01:59:13.142814 10238 heartbeater.cc:507] Master 127.0.71.124:43671 requested a full tablet report, sending...
I20260929 01:59:13.146175 9669 ts_manager.cc:194] Re-registered known tserver with Master: 25146d82715b417f826b236363302518 (127.0.71.65:37775)
I20260929 01:59:13.157176 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:59:13.157482 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 9486
I20260929 01:59:13.159332 9547 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:59:13.298084 10252 sys_catalog.cc:455] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "e647e46c807f4ad8abb25157f59fb74d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } } }
I20260929 01:59:13.298717 10252 sys_catalog.cc:458] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [sys.catalog]: This master's current role is: LEADER
I20260929 01:59:13.301087 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 9486
W20260929 01:59:13.325977 10047 connection.cc:573] server connection from 127.0.0.1:57026 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:59:13.327666 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:33791
--webserver_interface=127.0.71.126
--webserver_port=39359
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--rpc_reuseport=true
--master_addresses=127.0.71.126:33791,127.0.71.125:34895,127.0.71.124:43671
--master_addresses=127.0.71.126:33791,127.0.71.125:34895,127.0.71.124:43671,127.0.71.123:33165 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:59:13.781746 10256 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:59:13.782262 10256 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:59:13.782505 10256 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:59:13.792263 10256 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:59:13.792418 10256 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:59:13.792574 10256 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:59:13.792635 10256 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:59:13.805440 10256 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:33791,127.0.71.125:34895,127.0.71.124:43671,127.0.71.123:33165
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:33791
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=39359
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:59:13.807492 10256 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:59:13.809121 10256 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:59:13.819754 10261 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:13.820413 10264 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:13.820780 10262 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:59:13.821473 10256 server_base.cc:1061] running on GCE node
I20260929 01:59:13.822984 10256 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:59:13.823273 10256 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:59:13.823446 10256 hybrid_clock.cc:648] HybridClock initialized: now 1790647153823405 us; error 0 us; skew 500 ppm
I20260929 01:59:13.823871 10256 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:59:13 dist-test-slave-vz23 krb5kdc[9189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647153, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:59:13.829033 10256 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:59:13.831852 10256 webserver.cc:533] Webserver started at http://127.0.71.126:39359/ using document root <none> and password file <none>
I20260929 01:59:13.832927 10256 fs_manager.cc:362] Metadata directory not provided
I20260929 01:59:13.833097 10256 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:59:13.842420 10256 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.007s sys 0.000s
I20260929 01:59:13.847291 10270 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:13.849722 10256 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.002s sys 0.000s
I20260929 01:59:13.850021 10256 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "e647e46c807f4ad8abb25157f59fb74d"
format_stamp: "Formatted at 2026-09-29 01:59:08 on dist-test-slave-vz23"
I20260929 01:59:13.850977 10256 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:59:13.894446 10256 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:59:13.908124 10256 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:59:13.908661 10256 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:59:13.936178 10256 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:33791
I20260929 01:59:13.936146 10321 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:33791 every 8 connection(s)
I20260929 01:59:13.938445 10256 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:59:13.946692 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 10256
I20260929 01:59:13.947119 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 10033
I20260929 01:59:13.948166 10322 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:59:13.953918 10322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d: Bootstrap starting.
I20260929 01:59:13.969157 10232 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:59:13.980101 10322 log.cc:826] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d: Log is configured to *not* fsync() on all Append() calls
I20260929 01:59:13.993166 10322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:59:13.993873 10322 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d: Bootstrap complete.
I20260929 01:59:14.001978 10322 raft_consensus.cc:359] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } }
I20260929 01:59:14.002949 10322 raft_consensus.cc:740] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: e647e46c807f4ad8abb25157f59fb74d, State: Initialized, Role: FOLLOWER
I20260929 01:59:14.003907 10322 consensus_queue.cc:260] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } }
I20260929 01:59:14.004974 10328 sys_catalog.cc:455] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } } }
I20260929 01:59:14.005388 10328 sys_catalog.cc:458] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:14.006426 10322 sys_catalog.cc:565] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:59:14.019852 10339 catalog_manager.cc:1308] Loaded cluster ID: ed6b7a0757884b82802fb0d43c0d88c5
I20260929 01:59:14.020154 10339 catalog_manager.cc:1601] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d: loading cluster ID for follower catalog manager: success
I20260929 01:59:14.025509 10339 catalog_manager.cc:1623] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d: acquiring CA information for follower catalog manager: success
I20260929 01:59:14.029810 10339 catalog_manager.cc:1651] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:59:14.094117 10256 master_runner.cc:264] No leader master found from master e647e46c807f4ad8abb25157f59fb74d
I20260929 01:59:14.094390 10256 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:59:14.115121 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 10033
I20260929 01:59:14.144068 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:37775
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:33791,127.0.71.125:34895,127.0.71.124:43671
--webserver_port=36029
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--tserver_master_addrs=127.0.71.126:33791,127.0.71.125:34895,127.0.71.124:43671,127.0.71.123:33165 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:59:14.261092 10347 raft_consensus.cc:493] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e647e46c807f4ad8abb25157f59fb74d)
I20260929 01:59:14.261492 10347 raft_consensus.cc:515] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } }
I20260929 01:59:14.263561 10347 leader_election.cc:290] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e647e46c807f4ad8abb25157f59fb74d (127.0.71.126:33791), 8f7c23b6f007491f904a323c402755a1 (127.0.71.125:34895)
I20260929 01:59:14.265446 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae6a09befe714482a4fa52a23f43cf5d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "8f7c23b6f007491f904a323c402755a1" is_pre_election: true
I20260929 01:59:14.277808 10297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae6a09befe714482a4fa52a23f43cf5d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "e647e46c807f4ad8abb25157f59fb74d" is_pre_election: true
I20260929 01:59:14.278800 10297 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate ae6a09befe714482a4fa52a23f43cf5d in term 1.
I20260929 01:59:14.279829 9656 leader_election.cc:304] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae6a09befe714482a4fa52a23f43cf5d, e647e46c807f4ad8abb25157f59fb74d; no voters: 8f7c23b6f007491f904a323c402755a1
I20260929 01:59:14.280390 10347 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 01:59:14.280593 10347 raft_consensus.cc:493] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [term 1 FOLLOWER]: Starting leader election (detected failure of leader e647e46c807f4ad8abb25157f59fb74d)
I20260929 01:59:14.280699 10347 raft_consensus.cc:3060] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:59:14.284947 10347 raft_consensus.cc:515] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [term 2 FOLLOWER]: Starting leader election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } }
I20260929 01:59:14.286439 10347 leader_election.cc:290] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [CANDIDATE]: Term 2 election: Requested vote from peers e647e46c807f4ad8abb25157f59fb74d (127.0.71.126:33791), 8f7c23b6f007491f904a323c402755a1 (127.0.71.125:34895)
I20260929 01:59:14.286957 9601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae6a09befe714482a4fa52a23f43cf5d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "8f7c23b6f007491f904a323c402755a1"
I20260929 01:59:14.288021 10297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "ae6a09befe714482a4fa52a23f43cf5d" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "e647e46c807f4ad8abb25157f59fb74d"
I20260929 01:59:14.288307 10297 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:59:14.295222 10297 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate ae6a09befe714482a4fa52a23f43cf5d in term 2.
I20260929 01:59:14.296057 9656 leader_election.cc:304] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: ae6a09befe714482a4fa52a23f43cf5d, e647e46c807f4ad8abb25157f59fb74d; no voters: 8f7c23b6f007491f904a323c402755a1
I20260929 01:59:14.296831 10347 raft_consensus.cc:2804] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [term 2 FOLLOWER]: Leader election won for term 2
I20260929 01:59:14.297115 10349 raft_consensus.cc:493] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [term 1 FOLLOWER]: Starting pre-election (detected failure of leader e647e46c807f4ad8abb25157f59fb74d)
I20260929 01:59:14.297458 10349 raft_consensus.cc:515] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [term 1 FOLLOWER]: Starting pre-election with config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } }
I20260929 01:59:14.301666 10349 leader_election.cc:290] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers e647e46c807f4ad8abb25157f59fb74d (127.0.71.126:33791), ae6a09befe714482a4fa52a23f43cf5d (127.0.71.124:43671)
I20260929 01:59:14.301959 9678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f7c23b6f007491f904a323c402755a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "ae6a09befe714482a4fa52a23f43cf5d" is_pre_election: true
I20260929 01:59:14.303386 10347 raft_consensus.cc:697] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [term 2 LEADER]: Becoming Leader. State: Replica: ae6a09befe714482a4fa52a23f43cf5d, State: Running, Role: LEADER
I20260929 01:59:14.304104 10347 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } }
I20260929 01:59:14.314671 10350 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae6a09befe714482a4fa52a23f43cf5d. Latest consensus state: current_term: 2 leader_uuid: "ae6a09befe714482a4fa52a23f43cf5d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } } }
I20260929 01:59:14.315188 10350 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [sys.catalog]: This master's current role is: LEADER
I20260929 01:59:14.319249 10297 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "8f7c23b6f007491f904a323c402755a1" candidate_term: 2 candidate_status { last_received { term: 1 index: 5 } } ignore_live_leader: false dest_uuid: "e647e46c807f4ad8abb25157f59fb74d" is_pre_election: true
I20260929 01:59:14.319758 10297 raft_consensus.cc:2393] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [term 2 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8f7c23b6f007491f904a323c402755a1 in current term 2: Already voted for candidate ae6a09befe714482a4fa52a23f43cf5d in this term.
I20260929 01:59:14.320842 9579 leader_election.cc:304] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8f7c23b6f007491f904a323c402755a1; no voters: ae6a09befe714482a4fa52a23f43cf5d, e647e46c807f4ad8abb25157f59fb74d
I20260929 01:59:14.321300 10349 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [term 1 FOLLOWER]: Advancing to term 2
I20260929 01:59:14.325035 10349 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [term 2 FOLLOWER]: Leader pre-election lost for term 2. Reason: could not achieve majority
I20260929 01:59:14.326818 10353 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:59:14.329895 10353 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:59:14.331248 10353 catalog_manager.cc:1308] Loaded cluster ID: ed6b7a0757884b82802fb0d43c0d88c5
I20260929 01:59:14.331383 10353 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:59:14.333200 10353 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:59:14.334519 10353 catalog_manager.cc:6230] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d: Loaded TSK: 0
I20260929 01:59:14.335909 10353 catalog_manager.cc:1563] Initializing in-progress tserver states...
W20260929 01:59:14.635661 10346 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:59:14.636142 10346 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:59:14.636325 10346 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:59:14.645675 10346 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:59:14.645987 10346 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:59:14.658229 10346 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:37775
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=36029
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:33791,127.0.71.125:34895,127.0.71.124:43671,127.0.71.123:33165
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:59:14.660257 10346 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:59:14.662130 10346 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:59:14.675398 10361 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:14.675331 10358 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:14.675400 10359 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:59:14.676810 10346 server_base.cc:1061] running on GCE node
I20260929 01:59:14.677589 10346 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:59:14.677763 10346 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:59:14.677901 10346 hybrid_clock.cc:648] HybridClock initialized: now 1790647154677878 us; error 0 us; skew 500 ppm
I20260929 01:59:14.678315 10346 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:59:14 dist-test-slave-vz23 krb5kdc[9189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:59:14.682797 10346 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:59:14.685145 10346 webserver.cc:533] Webserver started at http://127.0.71.65:36029/ using document root <none> and password file <none>
I20260929 01:59:14.686098 10346 fs_manager.cc:362] Metadata directory not provided
I20260929 01:59:14.686261 10346 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:59:14.694160 10346 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 01:59:14.698535 10367 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:14.701604 10346 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.006s sys 0.000s
I20260929 01:59:14.701975 10346 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "25146d82715b417f826b236363302518"
format_stamp: "Formatted at 2026-09-29 01:59:11 on dist-test-slave-vz23"
I20260929 01:59:14.703474 10346 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:59:14.713924 10070 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84 [term 1 LEARNER]: Advancing to term 2
I20260929 01:59:14.717067 10070 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84 [term 2 LEARNER]: Refusing update from remote peer ae6a09befe714482a4fa52a23f43cf5d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20260929 01:59:14.718353 10350 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:59:14.722764 10373 sys_catalog.cc:455] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae6a09befe714482a4fa52a23f43cf5d. Latest consensus state: current_term: 2 leader_uuid: "ae6a09befe714482a4fa52a23f43cf5d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } } }
I20260929 01:59:14.723177 10373 sys_catalog.cc:458] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84 [sys.catalog]: This master's current role is: LEARNER
I20260929 01:59:14.724476 10350 raft_consensus.cc:1064] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d: attempting to promote NON_VOTER e33df7eb69674f6ba64c48539cd9aa84 to VOTER
W20260929 01:59:14.724977 10350 raft_consensus.cc:1068] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d: Unable to promote non-voter e33df7eb69674f6ba64c48539cd9aa84: Illegal state: Leader has not yet committed an operation in its own term
I20260929 01:59:14.728066 9601 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [term 2 FOLLOWER]: Refusing update from remote peer ae6a09befe714482a4fa52a23f43cf5d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20260929 01:59:14.729127 10350 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:59:14.731586 10349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae6a09befe714482a4fa52a23f43cf5d. Latest consensus state: current_term: 2 leader_uuid: "ae6a09befe714482a4fa52a23f43cf5d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } } }
I20260929 01:59:14.731945 10349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:14.732818 10346 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:59:14.735524 10297 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [term 2 FOLLOWER]: Refusing update from remote peer ae6a09befe714482a4fa52a23f43cf5d: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 2 index: 6. (index mismatch)
I20260929 01:59:14.736687 10350 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:59:14.741072 10346 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:59:14.741564 10346 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:59:14.742666 10349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ae6a09befe714482a4fa52a23f43cf5d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } } }
I20260929 01:59:14.743044 10349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:14.743424 10346 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:59:14.745949 10346 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:59:14.746098 10346 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:14.746227 10346 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:59:14.746326 10346 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:14.746546 10373 sys_catalog.cc:455] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ae6a09befe714482a4fa52a23f43cf5d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } } }
I20260929 01:59:14.746852 10373 sys_catalog.cc:458] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84 [sys.catalog]: This master's current role is: LEARNER
I20260929 01:59:14.748752 10386 sys_catalog.cc:455] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ae6a09befe714482a4fa52a23f43cf5d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } } }
I20260929 01:59:14.749107 10386 sys_catalog.cc:458] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:14.749644 10379 sys_catalog.cc:455] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [sys.catalog]: SysCatalogTable state changed. Reason: New leader ae6a09befe714482a4fa52a23f43cf5d. Latest consensus state: current_term: 2 leader_uuid: "ae6a09befe714482a4fa52a23f43cf5d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } } }
I20260929 01:59:14.749871 10379 sys_catalog.cc:458] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:14.750846 10347 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ae6a09befe714482a4fa52a23f43cf5d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } } }
I20260929 01:59:14.751195 10347 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [sys.catalog]: This master's current role is: LEADER
I20260929 01:59:14.751652 10387 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ae6a09befe714482a4fa52a23f43cf5d" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: true } } }
I20260929 01:59:14.751911 10377 raft_consensus.cc:1064] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d: attempting to promote NON_VOTER e33df7eb69674f6ba64c48539cd9aa84 to VOTER
I20260929 01:59:14.752075 10387 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [sys.catalog]: This master's current role is: LEADER
I20260929 01:59:14.753903 10377 consensus_queue.cc:237] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 6, Committed index: 6, Last appended: 2.6, Last appended by leader: 5, Current term: 2, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: false } }
I20260929 01:59:14.757529 10070 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84 [term 2 LEARNER]: Refusing update from remote peer ae6a09befe714482a4fa52a23f43cf5d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20260929 01:59:14.758096 9601 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [term 2 FOLLOWER]: Refusing update from remote peer ae6a09befe714482a4fa52a23f43cf5d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20260929 01:59:14.758761 10297 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [term 2 FOLLOWER]: Refusing update from remote peer ae6a09befe714482a4fa52a23f43cf5d: Log matching property violated. Preceding OpId in replica: term: 2 index: 6. Preceding OpId from leader: term: 2 index: 7. (index mismatch)
I20260929 01:59:14.759130 10387 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 01:59:14.760161 10347 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 01:59:14.761566 10347 consensus_queue.cc:1048] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [LEADER]: Connected to new peer: Peer: permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 7, Last known committed idx: 6, Time since last communication: 0.000s
I20260929 01:59:14.765159 10387 raft_consensus.cc:2955] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [term 2 LEADER]: Committing config change with OpId 2.7: config changed from index 5 to 7, e33df7eb69674f6ba64c48539cd9aa84 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: false } } }
I20260929 01:59:14.766520 10297 raft_consensus.cc:2955] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, e33df7eb69674f6ba64c48539cd9aa84 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: false } } }
I20260929 01:59:14.766996 9601 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, e33df7eb69674f6ba64c48539cd9aa84 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: false } } }
I20260929 01:59:14.771966 10379 sys_catalog.cc:455] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ae6a09befe714482a4fa52a23f43cf5d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: false } } }
I20260929 01:59:14.772269 10379 sys_catalog.cc:458] T 00000000000000000000000000000000 P e647e46c807f4ad8abb25157f59fb74d [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:14.773274 10387 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ae6a09befe714482a4fa52a23f43cf5d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: false } } }
I20260929 01:59:14.773550 10387 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [sys.catalog]: This master's current role is: LEADER
I20260929 01:59:14.773223 10070 raft_consensus.cc:2955] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84 [term 2 FOLLOWER]: Committing config change with OpId 2.7: config changed from index 5 to 7, e33df7eb69674f6ba64c48539cd9aa84 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: false } } }
I20260929 01:59:14.774466 10387 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ae6a09befe714482a4fa52a23f43cf5d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: false } } }
I20260929 01:59:14.774760 10387 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [sys.catalog]: This master's current role is: LEADER
I20260929 01:59:14.777004 10349 sys_catalog.cc:455] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ae6a09befe714482a4fa52a23f43cf5d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: false } } }
I20260929 01:59:14.777451 10349 sys_catalog.cc:458] T 00000000000000000000000000000000 P 8f7c23b6f007491f904a323c402755a1 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:14.777632 10347 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 2 leader_uuid: "ae6a09befe714482a4fa52a23f43cf5d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: false } } }
I20260929 01:59:14.777971 10347 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [sys.catalog]: This master's current role is: LEADER
I20260929 01:59:14.779172 10350 sys_catalog.cc:455] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 2 leader_uuid: "ae6a09befe714482a4fa52a23f43cf5d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: false } } }
I20260929 01:59:14.779461 10350 sys_catalog.cc:458] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d [sys.catalog]: This master's current role is: LEADER
I20260929 01:59:14.779902 10373 sys_catalog.cc:455] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 2 leader_uuid: "ae6a09befe714482a4fa52a23f43cf5d" committed_config { opid_index: 7 OBSOLETE_local: false peers { permanent_uuid: "e647e46c807f4ad8abb25157f59fb74d" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33791 } } peers { permanent_uuid: "8f7c23b6f007491f904a323c402755a1" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 34895 } } peers { permanent_uuid: "ae6a09befe714482a4fa52a23f43cf5d" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 43671 } } peers { permanent_uuid: "e33df7eb69674f6ba64c48539cd9aa84" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 33165 } attrs { promote: false } } }
I20260929 01:59:14.780184 10373 sys_catalog.cc:458] T 00000000000000000000000000000000 P e33df7eb69674f6ba64c48539cd9aa84 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:14.815739 10346 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:37775
I20260929 01:59:14.815814 10493 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:37775 every 8 connection(s)
I20260929 01:59:14.817692 10346 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:59:14.821534 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 10346
Sep 29 01:59:14 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647126, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for hive/127.0.0.1@KRBTEST.COM
Sep 29 01:59:14 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:59:14 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:59:14 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
Sep 29 01:59:14 dist-test-slave-vz23 krb5kdc[9189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647154, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:59:14.911458 10495 heartbeater.cc:344] Connected to a master server at 127.0.71.124:43671
I20260929 01:59:14.912016 10495 heartbeater.cc:461] Registering TS with master...
I20260929 01:59:14.913570 10495 heartbeater.cc:507] Master 127.0.71.124:43671 requested a full tablet report, sending...
I20260929 01:59:14.916319 10497 heartbeater.cc:344] Connected to a master server at 127.0.71.125:34895
I20260929 01:59:14.916553 10497 heartbeater.cc:461] Registering TS with master...
I20260929 01:59:14.916848 9669 ts_manager.cc:194] Re-registered known tserver with Master: 25146d82715b417f826b236363302518 (127.0.71.65:37775)
I20260929 01:59:14.917161 10497 heartbeater.cc:507] Master 127.0.71.125:34895 requested a full tablet report, sending...
I20260929 01:59:14.918869 9592 ts_manager.cc:194] Re-registered known tserver with Master: 25146d82715b417f826b236363302518 (127.0.71.65:37775)
I20260929 01:59:14.919261 9669 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:36537
I20260929 01:59:14.920372 10494 heartbeater.cc:344] Connected to a master server at 127.0.71.123:33165
I20260929 01:59:14.920578 10494 heartbeater.cc:461] Registering TS with master...
I20260929 01:59:14.921221 10494 heartbeater.cc:507] Master 127.0.71.123:33165 requested a full tablet report, sending...
I20260929 01:59:14.922953 10060 ts_manager.cc:194] Re-registered known tserver with Master: 25146d82715b417f826b236363302518 (127.0.71.65:37775)
I20260929 01:59:14.924943 10496 heartbeater.cc:344] Connected to a master server at 127.0.71.126:33791
I20260929 01:59:14.925171 10496 heartbeater.cc:461] Registering TS with master...
I20260929 01:59:14.926143 10496 heartbeater.cc:507] Master 127.0.71.126:33791 requested a full tablet report, sending...
I20260929 01:59:14.928855 10287 ts_manager.cc:194] Registered new tserver with Master: 25146d82715b417f826b236363302518 (127.0.71.65:37775)
I20260929 01:59:15.057039 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 10346
W20260929 01:59:15.059088 10511 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:59:15.054302 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:48796 (local address 127.0.71.65:37775)
0929 01:59:15.055062 (+ 760us) server_negotiation.cc:207] Beginning negotiation
0929 01:59:15.055069 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0929 01:59:15.055100 (+ 31us) server_negotiation.cc:408] Connection header received
0929 01:59:15.055170 (+ 70us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:59:15.055178 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:59:15.055424 (+ 246us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:59:15.055650 (+ 226us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:59:15.056861 (+ 1211us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:48796: BlockingRecv error: recv got EOF from 127.0.0.1:48796 (error 108)
Metrics: {"server-negotiator.queue_time_us":577,"thread_start_us":361,"threads_started":1}
W20260929 01:59:15.060812 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 10346: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:37775: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:59:15.061048 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 10346
I20260929 01:59:15.079789 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:37775
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:33791,127.0.71.125:34895,127.0.71.124:43671
--webserver_port=36029
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--tserver_master_addrs=127.0.71.126:33791,127.0.71.125:34895,127.0.71.124:43671,127.0.71.123:33165
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:59:15.505450 10513 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:59:15.505816 10513 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:59:15.506011 10513 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:59:15.506104 10513 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:59:15.515825 10513 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:59:15.516300 10513 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:59:15.530122 10513 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:37775
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=36029
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:33791,127.0.71.125:34895,127.0.71.124:43671,127.0.71.123:33165
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:59:15.532168 10513 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:59:15.533978 10513 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:59:15.550407 10519 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:15.550522 10521 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:15.550158 10518 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:59:15.551352 10513 server_base.cc:1061] running on GCE node
I20260929 01:59:15.552242 10513 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:59:15.552505 10513 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:59:15.552770 10513 hybrid_clock.cc:648] HybridClock initialized: now 1790647155552678 us; error 0 us; skew 500 ppm
I20260929 01:59:15.553380 10513 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:59:15 dist-test-slave-vz23 krb5kdc[9189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647155, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:59:15.558692 10513 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:59:15.561746 10513 webserver.cc:533] Webserver started at http://127.0.71.65:36029/ using document root <none> and password file <none>
I20260929 01:59:15.563117 10513 fs_manager.cc:362] Metadata directory not provided
I20260929 01:59:15.563283 10513 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:59:15.570863 10513 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:59:15.575151 10527 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:15.577553 10513 fs_manager.cc:730] Time spent opening block manager: real 0.005s user 0.003s sys 0.000s
I20260929 01:59:15.577884 10513 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "25146d82715b417f826b236363302518"
format_stamp: "Formatted at 2026-09-29 01:59:11 on dist-test-slave-vz23"
I20260929 01:59:15.578842 10513 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:59:15.604959 10513 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:59:15.610726 10513 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:59:15.611114 10513 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:59:15.612321 10513 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:59:15.614671 10513 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:59:15.614787 10513 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:15.614955 10513 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:59:15.615036 10513 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:59:15.615643 10513 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:59:15.615753 10513 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:59:15.615875 10513 fault_injection.cc:59] injecting fault for kudu (pid 10513): FLAGS_fault_before_start (process will exit)
I20260929 01:59:15.638000 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:37775
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:33791,127.0.71.125:34895,127.0.71.124:43671
--webserver_port=36029
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--time_source=system_unsync
--tserver_master_addrs=127.0.71.126:33791,127.0.71.125:34895,127.0.71.124:43671,127.0.71.123:33165 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:59:16.060776 10535 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:59:16.061239 10535 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:59:16.061439 10535 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:59:16.071367 10535 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:59:16.071655 10535 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:59:16.083806 10535 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--time_source=system_unsync
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:37775
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=36029
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:33791,127.0.71.125:34895,127.0.71.124:43671,127.0.71.123:33165
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:59:16.086185 10535 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:59:16.088037 10535 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:59:16.100605 10540 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:16.101743 10541 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:16.102923 10543 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:59:16.103335 10535 server_base.cc:1061] running on GCE node
I20260929 01:59:16.104226 10535 hybrid_clock.cc:584] initializing the hybrid clock with 'system_unsync' time source
W20260929 01:59:16.104501 10535 system_unsync_time.cc:38] NTP support is disabled. Clock error bounds will not be accurate. This configuration is not suitable for distributed clusters.
I20260929 01:59:16.104640 10535 hybrid_clock.cc:648] HybridClock initialized: now 1790647156104593 us; error 0 us; skew 500 ppm
I20260929 01:59:16.105091 10535 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:59:16 dist-test-slave-vz23 krb5kdc[9189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647156, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:59:16.109974 10535 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:59:16.112433 10535 webserver.cc:533] Webserver started at http://127.0.71.65:36029/ using document root <none> and password file <none>
I20260929 01:59:16.113471 10535 fs_manager.cc:362] Metadata directory not provided
I20260929 01:59:16.113656 10535 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:59:16.120857 10535 fs_manager.cc:714] Time spent opening directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 01:59:16.124034 10549 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:16.125735 10535 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:59:16.125988 10535 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "25146d82715b417f826b236363302518"
format_stamp: "Formatted at 2026-09-29 01:59:11 on dist-test-slave-vz23"
I20260929 01:59:16.126891 10535 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:59:16.154554 10535 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:59:16.159904 10535 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:59:16.160351 10535 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:59:16.161778 10535 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:59:16.164197 10535 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:59:16.164387 10535 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:16.164619 10535 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:59:16.164729 10535 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:16.213018 10535 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:37775
I20260929 01:59:16.213078 10661 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:37775 every 8 connection(s)
I20260929 01:59:16.215008 10535 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:59:16.225411 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 10535
I20260929 01:59:16.225811 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 10535
W20260929 01:59:16.258184 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 10535: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:37775: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:59:16.258332 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 10535
W20260929 01:59:16.278528 10669 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:59:16.221385 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:45087 (local address 127.0.71.124:43671)
0929 01:59:16.233011 (+ 11626us) server_negotiation.cc:207] Beginning negotiation
0929 01:59:16.233020 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0929 01:59:16.247897 (+ 14877us) server_negotiation.cc:408] Connection header received
0929 01:59:16.248113 (+ 216us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:59:16.248122 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:59:16.248230 (+ 108us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:59:16.248376 (+ 146us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:59:16.254791 (+ 6415us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:59:16.256358 (+ 1567us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:59:16.277554 (+ 21196us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:45087: BlockingRecv error: recv got EOF from 127.0.71.65:45087 (error 108)
Metrics: {"server-negotiator.queue_time_us":11449,"thread_start_us":11310,"threads_started":1}
W20260929 01:59:16.278553 10666 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:59:16.219792 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:46935 (local address 127.0.71.126:33791)
0929 01:59:16.251054 (+ 31262us) server_negotiation.cc:207] Beginning negotiation
0929 01:59:16.251060 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0929 01:59:16.253593 (+ 2533us) server_negotiation.cc:408] Connection header received
0929 01:59:16.253900 (+ 307us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:59:16.253907 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:59:16.254009 (+ 102us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:59:16.254136 (+ 127us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:59:16.256521 (+ 2385us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:59:16.260608 (+ 4087us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:59:16.277569 (+ 16961us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:46935: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":31164,"thread_start_us":31049,"threads_started":1}
W20260929 01:59:16.278553 10667 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:59:16.219787 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:39489 (local address 127.0.71.123:33165)
0929 01:59:16.248486 (+ 28699us) server_negotiation.cc:207] Beginning negotiation
0929 01:59:16.248492 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0929 01:59:16.248513 (+ 21us) server_negotiation.cc:408] Connection header received
0929 01:59:16.248586 (+ 73us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:59:16.248594 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:59:16.248679 (+ 85us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:59:16.248832 (+ 153us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:59:16.255519 (+ 6687us) server_negotiation.cc:366] Received TLS_HANDSHAKE NegotiatePB request
0929 01:59:16.257105 (+ 1586us) server_negotiation.cc:378] Sending TLS_HANDSHAKE NegotiatePB response
0929 01:59:16.277615 (+ 20510us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:39489: BlockingRecv error: recv got EOF from 127.0.71.65:39489 (error 108)
Metrics: {"server-negotiator.queue_time_us":28538,"thread_start_us":28397,"threads_started":1}
I20260929 01:59:16.279054 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 10256
W20260929 01:59:16.280584 10674 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:59:16.225179 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:51159 (local address 127.0.71.125:34895)
0929 01:59:16.250421 (+ 25242us) server_negotiation.cc:207] Beginning negotiation
0929 01:59:16.250426 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0929 01:59:16.254337 (+ 3911us) server_negotiation.cc:408] Connection header received
0929 01:59:16.254491 (+ 154us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:59:16.254497 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:59:16.254593 (+ 96us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:59:16.254719 (+ 126us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:59:16.279351 (+ 24632us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:51159: BlockingRecv error: recv got EOF from 127.0.71.65:51159 (error 108)
Metrics: {"server-negotiator.queue_time_us":25125,"thread_start_us":25012,"threads_started":1}
W20260929 01:59:16.281813 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 10256: Not authorized: Client connection negotiation failed: client connection to 127.0.71.126:33791: server requires authentication, but client does not have Kerberos credentials available
W20260929 01:59:16.281957 10666 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:59:16.280105 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:34938 (local address 127.0.71.126:33791)
0929 01:59:16.280304 (+ 199us) server_negotiation.cc:207] Beginning negotiation
0929 01:59:16.280310 (+ 6us) server_negotiation.cc:400] Waiting for connection header
0929 01:59:16.280335 (+ 25us) server_negotiation.cc:408] Connection header received
0929 01:59:16.280619 (+ 284us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:59:16.280627 (+ 8us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:59:16.280724 (+ 97us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:59:16.280838 (+ 114us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:59:16.281687 (+ 849us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:34938: BlockingRecv error: recv got EOF from 127.0.0.1:34938 (error 108)
Metrics: {"server-negotiator.queue_time_us":73}
I20260929 01:59:16.281976 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 10256
W20260929 01:59:16.305356 9656 connection.cc:573] client connection to 127.0.71.126:33791 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:59:16.305642 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 9561
W20260929 01:59:16.305662 9656 proxy.cc:239] Call had error, refreshing address and retrying: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 01:59:16.307071 9622 generic_service.cc:196] Checking for leaks (request via RPC)
W20260929 01:59:16.308103 9656 consensus_peers.cc:597] T 00000000000000000000000000000000 P ae6a09befe714482a4fa52a23f43cf5d -> Peer e647e46c807f4ad8abb25157f59fb74d (127.0.71.126:33791): Couldn't send request to peer e647e46c807f4ad8abb25157f59fb74d. Status: Network error: Client connection negotiation failed: client connection to 127.0.71.126:33791: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
I20260929 01:59:16.431402 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 9561
I20260929 01:59:16.454916 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 9638
I20260929 01:59:16.456355 9699 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:59:16.582376 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 9638
I20260929 01:59:16.605273 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 9915
I20260929 01:59:16.606760 10090 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:59:16.733361 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 9915
Shutting down hive metastore.
[ OK ] ExternalMiniClusterTest.TestAddMaster/9 (33296 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/10
Loading random data
Initializing database '/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:59:17 dist-test-slave-vz23 krb5kdc[10688](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:59:17 dist-test-slave-vz23 krb5kdc[10688](info): set up 2 sockets
Sep 29 01:59:17 dist-test-slave-vz23 krb5kdc[10688](info): commencing operation
krb5kdc: starting...
W20260929 01:59:19.378569 285 mini_kdc.cc:121] Time spent starting KDC: real 2.199s user 0.003s sys 0.037s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 01:59:19 dist-test-slave-vz23 krb5kdc[10688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:59:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:59:19Z Disabled control of system clock
WARNING: no policy specified for hive/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "hive/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/hive_127.0.0.1.keytab.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/hive_127.0.0.1.keytab.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/krb5.conf
2026-09-29 01:59:27: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
Sep 29 01:59:30 dist-test-slave-vz23 krb5kdc[10688](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790647170, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 01:59:36 dist-test-slave-vz23 krb5kdc[10688](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790647176, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 01:59:36 dist-test-slave-vz23 krb5kdc[10688](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790647176, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:59:38.364065 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:38413
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:44609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:38413,127.0.71.125:36257,127.0.71.124:36677 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:59:38.705384 10985 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:59:38.705679 10985 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:59:38.705762 10985 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:59:38.714182 10985 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:59:38.714282 10985 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:59:38.714349 10985 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:59:38.714388 10985 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:59:38.725502 10985 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:44609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:38413,127.0.71.125:36257,127.0.71.124:36677
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:38413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:59:38.727090 10985 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:59:38.728418 10985 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:59:38.737200 10991 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:38.737187 10990 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:38.737353 10993 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:59:38.737555 10985 server_base.cc:1061] running on GCE node
I20260929 01:59:38.738759 10985 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:59:38.740247 10985 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:59:38.741475 10985 hybrid_clock.cc:648] HybridClock initialized: now 1790647178741424 us; error 78 us; skew 500 ppm
I20260929 01:59:38.741926 10985 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:59:38 dist-test-slave-vz23 krb5kdc[10688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:59:38.745306 10985 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:59:38.747212 10985 webserver.cc:533] Webserver started at http://127.0.71.126:39281/ using document root <none> and password file <none>
I20260929 01:59:38.748010 10985 fs_manager.cc:362] Metadata directory not provided
I20260929 01:59:38.748145 10985 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:59:38.748605 10985 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:59:38.751199 10985 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "93831b88e43f471dbf8af58a90aca3c9"
format_stamp: "Formatted at 2026-09-29 01:59:38 on dist-test-slave-vz23"
I20260929 01:59:38.752066 10985 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "93831b88e43f471dbf8af58a90aca3c9"
format_stamp: "Formatted at 2026-09-29 01:59:38 on dist-test-slave-vz23"
I20260929 01:59:38.758263 10985 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.009s sys 0.000s
I20260929 01:59:38.762071 11000 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:38.763715 10985 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:59:38.764034 10985 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "93831b88e43f471dbf8af58a90aca3c9"
format_stamp: "Formatted at 2026-09-29 01:59:38 on dist-test-slave-vz23"
I20260929 01:59:38.764241 10985 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:59:38.783205 10985 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:59:38.787329 10985 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:59:38.787649 10985 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:59:38.810307 10985 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:38413
I20260929 01:59:38.810341 11051 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:38413 every 8 connection(s)
I20260929 01:59:38.812026 10985 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:59:38.816303 11052 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:59:38.820340 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 10985
I20260929 01:59:38.820555 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 01:59:38.822361 11052 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:59:38 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.125@KRBTEST.COM; defaulting to no policy
I20260929 01:59:38.846833 11052 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:59:38.848042 11003 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.125:36257: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.125@KRBTEST.COM" created.
W20260929 01:59:38.849957 11052 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:36257: Network error: Client connection negotiation failed: client connection to 127.0.71.125:36257: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:59:38.899336 11052 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } attempt: 1
WARNING: no policy specified for HTTP/127.0.71.125@KRBTEST.COM; defaulting to no policy
W20260929 01:59:38.901500 11052 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:36257: Network error: Client connection negotiation failed: client connection to 127.0.71.125:36257: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:59:38.937335 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:36257
--webserver_interface=127.0.71.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:44609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:38413,127.0.71.125:36257,127.0.71.124:36677 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:59:38.969872 11052 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } attempt: 2
W20260929 01:59:38.972035 11052 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:36257: Network error: Client connection negotiation failed: client connection to 127.0.71.125:36257: connect: Connection refused (error 111)
I20260929 01:59:39.063383 11052 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } attempt: 3
W20260929 01:59:39.065761 11052 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:36257: Network error: Client connection negotiation failed: client connection to 127.0.71.125:36257: connect: Connection refused (error 111)
I20260929 01:59:39.209051 11052 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } attempt: 4
W20260929 01:59:39.211160 11052 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:36257: Network error: Client connection negotiation failed: client connection to 127.0.71.125:36257: connect: Connection refused (error 111)
W20260929 01:59:39.276845 11061 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:59:39.277269 11061 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:59:39.277427 11061 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:59:39.286155 11061 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:59:39.286321 11061 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:59:39.286453 11061 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:59:39.286546 11061 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:59:39.297478 11061 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:44609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:38413,127.0.71.125:36257,127.0.71.124:36677
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:36257
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:59:39.299043 11061 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:59:39.300323 11061 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:59:39.309180 11067 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:39.309715 11066 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:39.309944 11069 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:59:39.310454 11061 server_base.cc:1061] running on GCE node
I20260929 01:59:39.311156 11061 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:59:39.312539 11061 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:59:39.313746 11061 hybrid_clock.cc:648] HybridClock initialized: now 1790647179313678 us; error 60 us; skew 500 ppm
I20260929 01:59:39.314085 11061 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:59:39 dist-test-slave-vz23 krb5kdc[10688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:59:39.317305 11061 init.cc:377] Logged in from keytab as kudu/127.0.71.125@KRBTEST.COM (short username kudu)
I20260929 01:59:39.318979 11061 webserver.cc:533] Webserver started at http://127.0.71.125:39955/ using document root <none> and password file <none>
I20260929 01:59:39.319664 11061 fs_manager.cc:362] Metadata directory not provided
I20260929 01:59:39.319787 11061 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:59:39.320130 11061 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:59:39.322561 11061 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-1/data/instance:
uuid: "c40cd5c3ac114930a92ebe1d974dbe1b"
format_stamp: "Formatted at 2026-09-29 01:59:39 on dist-test-slave-vz23"
I20260929 01:59:39.323345 11061 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-1/wal/instance:
uuid: "c40cd5c3ac114930a92ebe1d974dbe1b"
format_stamp: "Formatted at 2026-09-29 01:59:39 on dist-test-slave-vz23"
I20260929 01:59:39.328852 11061 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260929 01:59:39.332149 11076 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:39.333545 11061 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:59:39.333720 11061 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "c40cd5c3ac114930a92ebe1d974dbe1b"
format_stamp: "Formatted at 2026-09-29 01:59:39 on dist-test-slave-vz23"
I20260929 01:59:39.333914 11061 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:59:39.352157 11061 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:59:39.355836 11061 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:59:39.356117 11061 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:59:39.376991 11061 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:36257
I20260929 01:59:39.377076 11127 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:36257 every 8 connection(s)
I20260929 01:59:39.378438 11061 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 01:59:39.382761 11128 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:59:39.382797 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 11061
I20260929 01:59:39.382969 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 01:59:39.391696 11128 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:59:39 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.124@KRBTEST.COM; defaulting to no policy
I20260929 01:59:39.413218 11128 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.124@KRBTEST.COM" created.
Sep 29 01:59:39 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:59:39.428496 11128 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:59:39.429584 11078 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36677: connect: Connection refused (error 111)
W20260929 01:59:39.431234 11128 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36677: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36677: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.124@KRBTEST.COM" created.
I20260929 01:59:39.480563 11128 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } attempt: 1
W20260929 01:59:39.482560 11128 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36677: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36677: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:59:39.501739 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:36677
--webserver_interface=127.0.71.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:44609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:38413,127.0.71.125:36257,127.0.71.124:36677 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:59:39.510397 11052 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } attempt: 5
Sep 29 01:59:39 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:59:39.523468 11052 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } has no permanent_uuid. Determining permanent_uuid...
W20260929 01:59:39.525719 11052 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36677: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36677: connect: Connection refused (error 111)
I20260929 01:59:39.550828 11128 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } attempt: 2
W20260929 01:59:39.552786 11128 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36677: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36677: connect: Connection refused (error 111)
I20260929 01:59:39.576956 11052 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } attempt: 1
W20260929 01:59:39.579392 11052 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36677: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36677: connect: Connection refused (error 111)
I20260929 01:59:39.644062 11128 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } attempt: 3
W20260929 01:59:39.646168 11128 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36677: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36677: connect: Connection refused (error 111)
I20260929 01:59:39.647629 11052 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } attempt: 2
W20260929 01:59:39.649539 11052 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36677: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36677: connect: Connection refused (error 111)
I20260929 01:59:39.755807 11052 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } attempt: 3
W20260929 01:59:39.758080 11052 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36677: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36677: connect: Connection refused (error 111)
I20260929 01:59:39.789439 11128 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } attempt: 4
W20260929 01:59:39.791471 11128 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36677: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36677: connect: Connection refused (error 111)
W20260929 01:59:39.837535 11138 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:59:39.837833 11138 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:59:39.837989 11138 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:59:39.846526 11138 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:59:39.846679 11138 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:59:39.846798 11138 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:59:39.846879 11138 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:59:39.857754 11138 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:44609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:38413,127.0.71.125:36257,127.0.71.124:36677
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:36677
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:59:39.859426 11138 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:59:39.860819 11138 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:59:39.869050 11146 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:39.868994 11144 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:39.868994 11143 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:59:39.870260 11138 server_base.cc:1061] running on GCE node
I20260929 01:59:39.870980 11138 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:59:39.872236 11138 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:59:39.873442 11138 hybrid_clock.cc:648] HybridClock initialized: now 1790647179873387 us; error 49 us; skew 500 ppm
I20260929 01:59:39.873817 11138 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:59:39 dist-test-slave-vz23 krb5kdc[10688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:59:39.876942 11138 init.cc:377] Logged in from keytab as kudu/127.0.71.124@KRBTEST.COM (short username kudu)
I20260929 01:59:39.878479 11138 webserver.cc:533] Webserver started at http://127.0.71.124:40329/ using document root <none> and password file <none>
I20260929 01:59:39.879153 11138 fs_manager.cc:362] Metadata directory not provided
I20260929 01:59:39.879273 11138 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:59:39.879626 11138 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:59:39.882027 11138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-2/data/instance:
uuid: "7878d5fe172a493a8f052281b39a28b7"
format_stamp: "Formatted at 2026-09-29 01:59:39 on dist-test-slave-vz23"
I20260929 01:59:39.882803 11138 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-2/wal/instance:
uuid: "7878d5fe172a493a8f052281b39a28b7"
format_stamp: "Formatted at 2026-09-29 01:59:39 on dist-test-slave-vz23"
I20260929 01:59:39.888118 11138 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 01:59:39.891464 11153 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:39.892848 11138 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:59:39.893011 11138 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "7878d5fe172a493a8f052281b39a28b7"
format_stamp: "Formatted at 2026-09-29 01:59:39 on dist-test-slave-vz23"
I20260929 01:59:39.893193 11138 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:59:39.914229 11138 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:59:39.918100 11138 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:59:39.918385 11138 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:59:39.935352 11052 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } attempt: 4
W20260929 01:59:39.937772 11052 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:36677: Network error: Client connection negotiation failed: client connection to 127.0.71.124:36677: connect: Connection refused (error 111)
I20260929 01:59:39.939492 11138 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:36677
I20260929 01:59:39.939535 11204 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:36677 every 8 connection(s)
I20260929 01:59:39.941061 11138 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 01:59:39.945144 11205 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:59:39.947515 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 11138
I20260929 01:59:39.947729 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 01:59:39.952005 11205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:59:39 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:59:39 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:59:39 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:59:39.973780 11205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:59:39 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:59:39.988399 11205 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:59:39 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:59:39 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:59:40.006983 11205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7: Bootstrap starting.
I20260929 01:59:40.010206 11205 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7: Neither blocks nor log segments found. Creating new log.
I20260929 01:59:40.011394 11205 log.cc:826] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7: Log is configured to *not* fsync() on all Append() calls
I20260929 01:59:40.014137 11205 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7: No bootstrap required, opened a new log
I20260929 01:59:40.019315 11205 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } }
I20260929 01:59:40.019577 11205 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:59:40.019712 11205 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7878d5fe172a493a8f052281b39a28b7, State: Initialized, Role: FOLLOWER
I20260929 01:59:40.020293 11205 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } }
I20260929 01:59:40.021216 11215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } }
I20260929 01:59:40.021507 11215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:40.022198 11205 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:59:40.033831 11226 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:59:40.034042 11226 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:59:40.030128 11138 master_runner.cc:187] Error getting master registration for 127.0.71.126:38413: OK, instance_id { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" instance_seqno: 1790647178804084 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:59:40.036283 11138 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 01:59:40.090740 11128 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } attempt: 5
Sep 29 01:59:40 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:59:40.112128 11128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b: Bootstrap starting.
I20260929 01:59:40.116629 11128 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b: Neither blocks nor log segments found. Creating new log.
I20260929 01:59:40.118031 11128 log.cc:826] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b: Log is configured to *not* fsync() on all Append() calls
I20260929 01:59:40.121250 11128 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b: No bootstrap required, opened a new log
I20260929 01:59:40.129843 11128 raft_consensus.cc:359] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } }
I20260929 01:59:40.130199 11128 raft_consensus.cc:385] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:59:40.130330 11128 raft_consensus.cc:740] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c40cd5c3ac114930a92ebe1d974dbe1b, State: Initialized, Role: FOLLOWER
I20260929 01:59:40.131081 11128 consensus_queue.cc:260] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } }
I20260929 01:59:40.132069 11229 sys_catalog.cc:455] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } }
I20260929 01:59:40.132349 11229 sys_catalog.cc:458] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:40.133522 11128 sys_catalog.cc:565] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:59:40.144543 11240 catalog_manager.cc:1607] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:59:40.144781 11240 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:59:40.141444 11061 master_runner.cc:187] Error getting master registration for 127.0.71.126:38413: OK, instance_id { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" instance_seqno: 1790647178804084 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 01:59:40.148478 11061 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:59:40.170529 11159 tablet.cc:2433] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:59:40.215027 11052 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } attempt: 5
Sep 29 01:59:40 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647178, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:59:40.238433 11052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9: Bootstrap starting.
I20260929 01:59:40.242616 11052 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9: Neither blocks nor log segments found. Creating new log.
I20260929 01:59:40.243999 11052 log.cc:826] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9: Log is configured to *not* fsync() on all Append() calls
I20260929 01:59:40.247249 11052 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9: No bootstrap required, opened a new log
I20260929 01:59:40.254921 11052 raft_consensus.cc:359] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } }
I20260929 01:59:40.255255 11052 raft_consensus.cc:385] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:59:40.255446 11052 raft_consensus.cc:740] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93831b88e43f471dbf8af58a90aca3c9, State: Initialized, Role: FOLLOWER
I20260929 01:59:40.257934 11052 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } }
I20260929 01:59:40.258968 11243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } }
I20260929 01:59:40.259339 11243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:40.260212 11052 sys_catalog.cc:565] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:59:40.287194 10985 master_runner.cc:264] No leader master found from master 93831b88e43f471dbf8af58a90aca3c9
I20260929 01:59:40.287369 10985 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 01:59:40.287288 11254 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:59:40.287446 11254 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
W20260929 01:59:40.290900 11006 tablet.cc:2433] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 01:59:40.358938 11082 tablet.cc:2433] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 01:59:40.380892 11229 raft_consensus.cc:493] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:59:40.381162 11229 raft_consensus.cc:515] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } }
I20260929 01:59:40.382675 11243 raft_consensus.cc:493] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:59:40.382913 11243 raft_consensus.cc:515] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } }
I20260929 01:59:40.383046 11229 leader_election.cc:290] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93831b88e43f471dbf8af58a90aca3c9 (127.0.71.126:38413), 7878d5fe172a493a8f052281b39a28b7 (127.0.71.124:36677)
I20260929 01:59:40.383836 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93831b88e43f471dbf8af58a90aca3c9" is_pre_election: true
I20260929 01:59:40.384262 11027 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c40cd5c3ac114930a92ebe1d974dbe1b in term 0.
I20260929 01:59:40.384532 11243 leader_election.cc:290] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers c40cd5c3ac114930a92ebe1d974dbe1b (127.0.71.125:36257), 7878d5fe172a493a8f052281b39a28b7 (127.0.71.124:36677)
I20260929 01:59:40.385162 11215 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 01:59:40.385423 11215 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } }
I20260929 01:59:40.386910 11077 leader_election.cc:304] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93831b88e43f471dbf8af58a90aca3c9, c40cd5c3ac114930a92ebe1d974dbe1b; no voters:
I20260929 01:59:40.387315 11229 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 01:59:40.387471 11229 raft_consensus.cc:493] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 01:59:40.387565 11229 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:59:40.387853 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93831b88e43f471dbf8af58a90aca3c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" is_pre_election: true
I20260929 01:59:40.391239 11229 raft_consensus.cc:515] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } }
I20260929 01:59:40.391923 11103 raft_consensus.cc:2393] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 93831b88e43f471dbf8af58a90aca3c9 in current term 1: Already voted for candidate c40cd5c3ac114930a92ebe1d974dbe1b in this term.
I20260929 01:59:40.388530 11180 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "93831b88e43f471dbf8af58a90aca3c9" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7878d5fe172a493a8f052281b39a28b7" is_pre_election: true
I20260929 01:59:40.393965 11180 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 93831b88e43f471dbf8af58a90aca3c9 in term 0.
I20260929 01:59:40.394532 11002 leader_election.cc:304] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 7878d5fe172a493a8f052281b39a28b7, 93831b88e43f471dbf8af58a90aca3c9; no voters: c40cd5c3ac114930a92ebe1d974dbe1b
I20260929 01:59:40.394913 11027 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93831b88e43f471dbf8af58a90aca3c9"
I20260929 01:59:40.395100 11027 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:59:40.396608 11179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7878d5fe172a493a8f052281b39a28b7" is_pre_election: true
I20260929 01:59:40.397023 11179 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate c40cd5c3ac114930a92ebe1d974dbe1b in term 0.
I20260929 01:59:40.397360 11215 leader_election.cc:290] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 93831b88e43f471dbf8af58a90aca3c9 (127.0.71.126:38413), c40cd5c3ac114930a92ebe1d974dbe1b (127.0.71.125:36257)
I20260929 01:59:40.398499 11027 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c40cd5c3ac114930a92ebe1d974dbe1b in term 1.
I20260929 01:59:40.398672 11243 raft_consensus.cc:2764] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260929 01:59:40.399945 11103 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7878d5fe172a493a8f052281b39a28b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" is_pre_election: true
I20260929 01:59:40.400190 11103 raft_consensus.cc:2393] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 7878d5fe172a493a8f052281b39a28b7 in current term 1: Already voted for candidate c40cd5c3ac114930a92ebe1d974dbe1b in this term.
I20260929 01:59:40.401342 11026 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "7878d5fe172a493a8f052281b39a28b7" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "93831b88e43f471dbf8af58a90aca3c9" is_pre_election: true
I20260929 01:59:40.402308 11154 leader_election.cc:304] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 7878d5fe172a493a8f052281b39a28b7; no voters: 93831b88e43f471dbf8af58a90aca3c9, c40cd5c3ac114930a92ebe1d974dbe1b
I20260929 01:59:40.402990 11215 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [term 0 FOLLOWER]: Advancing to term 1
I20260929 01:59:40.406292 11215 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 01:59:40.407027 11077 leader_election.cc:304] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 93831b88e43f471dbf8af58a90aca3c9, c40cd5c3ac114930a92ebe1d974dbe1b; no voters:
I20260929 01:59:40.409257 11179 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "7878d5fe172a493a8f052281b39a28b7"
I20260929 01:59:40.412612 11179 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate c40cd5c3ac114930a92ebe1d974dbe1b in term 1.
I20260929 01:59:40.413355 11229 leader_election.cc:290] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [CANDIDATE]: Term 1 election: Requested vote from peers 93831b88e43f471dbf8af58a90aca3c9 (127.0.71.126:38413), 7878d5fe172a493a8f052281b39a28b7 (127.0.71.124:36677)
I20260929 01:59:40.413805 11229 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 01:59:40.414361 11229 raft_consensus.cc:697] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [term 1 LEADER]: Becoming Leader. State: Replica: c40cd5c3ac114930a92ebe1d974dbe1b, State: Running, Role: LEADER
I20260929 01:59:40.415158 11229 consensus_queue.cc:237] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } }
I20260929 01:59:40.434604 11256 sys_catalog.cc:455] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader c40cd5c3ac114930a92ebe1d974dbe1b. Latest consensus state: current_term: 1 leader_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } }
I20260929 01:59:40.435082 11256 sys_catalog.cc:458] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: This master's current role is: LEADER
I20260929 01:59:40.439720 11258 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 01:59:40.442230 11258 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 01:59:40.462368 11179 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [term 1 FOLLOWER]: Refusing update from remote peer c40cd5c3ac114930a92ebe1d974dbe1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:59:40.466480 11256 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:59:40.474663 11027 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [term 1 FOLLOWER]: Refusing update from remote peer c40cd5c3ac114930a92ebe1d974dbe1b: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 01:59:40.475643 11256 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 01:59:40.504004 11215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c40cd5c3ac114930a92ebe1d974dbe1b. Latest consensus state: current_term: 1 leader_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } }
I20260929 01:59:40.504312 11215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:40.509141 11259 mvcc.cc:204] Tried to move back new op lower bound from 7334490851160612864 to 7334490850992246784. Current Snapshot: MvccSnapshot[applied={T|T < 7334490851160612864}]
I20260929 01:59:40.514482 11243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c40cd5c3ac114930a92ebe1d974dbe1b. Latest consensus state: current_term: 1 leader_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } }
I20260929 01:59:40.514715 11243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:40.520974 11256 sys_catalog.cc:455] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } }
I20260929 01:59:40.521219 11256 sys_catalog.cc:458] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: This master's current role is: LEADER
I20260929 01:59:40.521503 11256 sys_catalog.cc:455] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } }
I20260929 01:59:40.521703 11256 sys_catalog.cc:458] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: This master's current role is: LEADER
I20260929 01:59:40.522984 11258 catalog_manager.cc:1396] Generated new cluster ID: 976be3b2fe9c4e609f2f1aaf1e0dc177
I20260929 01:59:40.523078 11258 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 01:59:40.528326 11243 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } }
I20260929 01:59:40.528553 11243 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:40.529942 11261 mvcc.cc:204] Tried to move back new op lower bound from 7334490851160612864 to 7334490850992246784. Current Snapshot: MvccSnapshot[applied={T|T < 7334490851160612864}]
I20260929 01:59:40.534493 11215 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } }
I20260929 01:59:40.534718 11215 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:40.621186 11258 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 01:59:40.623149 11258 catalog_manager.cc:1553] Loading token signing keys...
I20260929 01:59:40.641165 11258 catalog_manager.cc:6219] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b: Generated new TSK 0
I20260929 01:59:40.642356 11258 catalog_manager.cc:1563] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:59:40.800527 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:38413,127.0.71.125:36257,127.0.71.124:36677
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:44609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:59:41.036561 11226 catalog_manager.cc:1308] Loaded cluster ID: 976be3b2fe9c4e609f2f1aaf1e0dc177
I20260929 01:59:41.036705 11226 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7: loading cluster ID for follower catalog manager: success
I20260929 01:59:41.039886 11226 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7: acquiring CA information for follower catalog manager: success
I20260929 01:59:41.041708 11226 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 01:59:41.157745 11272 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:59:41.158145 11272 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:59:41.158331 11272 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:59:41.166983 11272 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:59:41.167232 11272 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:59:41.178359 11272 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:44609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:38413,127.0.71.125:36257,127.0.71.124:36677
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:59:41.179991 11272 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:59:41.181365 11272 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:59:41.191344 11277 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:41.191316 11278 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:41.192341 11280 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:59:41.193203 11272 server_base.cc:1061] running on GCE node
I20260929 01:59:41.193737 11272 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:59:41.194705 11272 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:59:41.195917 11272 hybrid_clock.cc:648] HybridClock initialized: now 1790647181195859 us; error 49 us; skew 500 ppm
I20260929 01:59:41.196199 11272 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:59:41 dist-test-slave-vz23 krb5kdc[10688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:59:41.199651 11272 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:59:41.201339 11272 webserver.cc:533] Webserver started at http://127.0.71.65:32989/ using document root <none> and password file <none>
I20260929 01:59:41.201992 11272 fs_manager.cc:362] Metadata directory not provided
I20260929 01:59:41.202095 11272 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:59:41.202373 11272 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:59:41.204763 11272 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "4a016aaa46334430a446bf502f0a94ff"
format_stamp: "Formatted at 2026-09-29 01:59:41 on dist-test-slave-vz23"
I20260929 01:59:41.205457 11272 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "4a016aaa46334430a446bf502f0a94ff"
format_stamp: "Formatted at 2026-09-29 01:59:41 on dist-test-slave-vz23"
I20260929 01:59:41.210673 11272 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 01:59:41.214018 11287 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:41.215606 11272 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 01:59:41.215790 11272 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "4a016aaa46334430a446bf502f0a94ff"
format_stamp: "Formatted at 2026-09-29 01:59:41 on dist-test-slave-vz23"
I20260929 01:59:41.215984 11272 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:59:41.252677 11272 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:59:41.257120 11272 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:59:41.257479 11272 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:59:41.258571 11272 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:59:41.260639 11272 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:59:41.260725 11272 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:41.260876 11272 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:59:41.260955 11272 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:41.290023 11254 catalog_manager.cc:1308] Loaded cluster ID: 976be3b2fe9c4e609f2f1aaf1e0dc177
I20260929 01:59:41.290155 11254 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9: loading cluster ID for follower catalog manager: success
I20260929 01:59:41.293820 11254 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9: acquiring CA information for follower catalog manager: success
I20260929 01:59:41.295655 11254 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:59:41.305827 11272 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:38977
I20260929 01:59:41.305910 11399 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:38977 every 8 connection(s)
I20260929 01:59:41.307312 11272 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:59:41.311776 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 11272
I20260929 01:59:41.311957 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
Sep 29 01:59:41 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:59:41 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:59:41 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:59:41.349789 11400 heartbeater.cc:344] Connected to a master server at 127.0.71.124:36677
I20260929 01:59:41.349808 11402 heartbeater.cc:344] Connected to a master server at 127.0.71.125:36257
I20260929 01:59:41.350260 11400 heartbeater.cc:461] Registering TS with master...
I20260929 01:59:41.350261 11402 heartbeater.cc:461] Registering TS with master...
I20260929 01:59:41.351496 11402 heartbeater.cc:507] Master 127.0.71.125:36257 requested a full tablet report, sending...
I20260929 01:59:41.351608 11401 heartbeater.cc:344] Connected to a master server at 127.0.71.126:38413
I20260929 01:59:41.351758 11401 heartbeater.cc:461] Registering TS with master...
I20260929 01:59:41.352181 11401 heartbeater.cc:507] Master 127.0.71.126:38413 requested a full tablet report, sending...
I20260929 01:59:41.352555 11400 heartbeater.cc:507] Master 127.0.71.124:36677 requested a full tablet report, sending...
I20260929 01:59:41.354199 11092 ts_manager.cc:194] Registered new tserver with Master: 4a016aaa46334430a446bf502f0a94ff (127.0.71.65:38977)
I20260929 01:59:41.354220 11017 ts_manager.cc:194] Registered new tserver with Master: 4a016aaa46334430a446bf502f0a94ff (127.0.71.65:38977)
I20260929 01:59:41.355460 11169 ts_manager.cc:194] Registered new tserver with Master: 4a016aaa46334430a446bf502f0a94ff (127.0.71.65:38977)
I20260929 01:59:41.356127 11092 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:45923
I20260929 01:59:41.371225 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
WARNING: no policy specified for kudu/127.0.71.123@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.123@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.123 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.123 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.123@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.123@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.123 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.123 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 01:59:41.488720 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.123:42445
--webserver_interface=127.0.71.123
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.123
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:44609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:38413,127.0.71.125:36257,127.0.71.124:36677,127.0.71.123:42445 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:59:41.824750 11414 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:59:41.825197 11414 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:59:41.825344 11414 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:59:41.833846 11414 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:59:41.833985 11414 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:59:41.834117 11414 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:59:41.834201 11414 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:59:41.845485 11414 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:44609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:38413,127.0.71.125:36257,127.0.71.124:36677,127.0.71.123:42445
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.123
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.123:42445
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.0.71.123
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:59:41.847606 11414 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:59:41.849188 11414 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:59:41.857220 11419 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:41.857255 11420 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:59:41.860718 11414 server_base.cc:1061] running on GCE node
W20260929 01:59:41.861897 11422 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:59:41.862789 11414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:59:41.864266 11414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:59:41.865473 11414 hybrid_clock.cc:648] HybridClock initialized: now 1790647181865386 us; error 79 us; skew 500 ppm
I20260929 01:59:41.865887 11414 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:59:41 dist-test-slave-vz23 krb5kdc[10688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:59:41.869194 11414 init.cc:377] Logged in from keytab as kudu/127.0.71.123@KRBTEST.COM (short username kudu)
I20260929 01:59:41.871172 11414 webserver.cc:533] Webserver started at http://127.0.71.123:40927/ using document root <none> and password file <none>
I20260929 01:59:41.871891 11414 fs_manager.cc:362] Metadata directory not provided
I20260929 01:59:41.871984 11414 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:59:41.872280 11414 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 01:59:41.874764 11414 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-3/data/instance:
uuid: "4316b0d33e304556b18eb80895390ee6"
format_stamp: "Formatted at 2026-09-29 01:59:41 on dist-test-slave-vz23"
I20260929 01:59:41.875571 11414 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-3/wal/instance:
uuid: "4316b0d33e304556b18eb80895390ee6"
format_stamp: "Formatted at 2026-09-29 01:59:41 on dist-test-slave-vz23"
I20260929 01:59:41.881081 11414 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 01:59:41.884421 11429 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:41.885867 11414 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:59:41.886010 11414 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "4316b0d33e304556b18eb80895390ee6"
format_stamp: "Formatted at 2026-09-29 01:59:41 on dist-test-slave-vz23"
I20260929 01:59:41.886197 11414 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:59:41.898442 11414 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:59:41.902302 11414 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:59:41.902526 11414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:59:41.923797 11414 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.123:42445
I20260929 01:59:41.923846 11480 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.123:42445 every 8 connection(s)
I20260929 01:59:41.925417 11414 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
I20260929 01:59:41.929529 11481 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 01:59:41.934131 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 11414
I20260929 01:59:41.935778 11481 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:59:41 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:59:41 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
I20260929 01:59:41.957247 11481 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:59:41 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:59:41.970499 11481 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:59:41 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:59:41.983388 11481 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42445 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 01:59:41 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647181, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
I20260929 01:59:42.000234 11481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6: Bootstrap starting.
I20260929 01:59:42.003605 11481 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6: Neither blocks nor log segments found. Creating new log.
I20260929 01:59:42.004756 11481 log.cc:826] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6: Log is configured to *not* fsync() on all Append() calls
I20260929 01:59:42.007225 11481 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6: No bootstrap required, opened a new log
I20260929 01:59:42.012605 11481 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42445 } }
I20260929 01:59:42.012941 11481 raft_consensus.cc:385] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 01:59:42.013055 11481 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 4316b0d33e304556b18eb80895390ee6, State: Initialized, Role: FOLLOWER
I20260929 01:59:42.013679 11481 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42445 } }
I20260929 01:59:42.014492 11490 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42445 } } }
I20260929 01:59:42.014794 11490 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:42.015434 11481 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 01:59:42.024765 11501 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 01:59:42.024920 11501 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 01:59:42.033169 11414 master_runner.cc:429] Detected that this master 4316b0d33e304556b18eb80895390ee6 is joining an existing cluster
I20260929 01:59:42.033310 11414 catalog_manager.cc:1037] Shutting down the Catalog Manager
I20260929 01:59:42.034168 11414 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260929 01:59:42.034554 11414 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260929 01:59:42.034785 11414 tablet_replica.cc:333] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6: stopping tablet replica
I20260929 01:59:42.036720 11414 master_runner.cc:434] Initiating AddMaster RPC to add 127.0.71.123:42445
Sep 29 01:59:42 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647179, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
I20260929 01:59:42.092938 11092 catalog_manager.cc:7351] Initiating ChangeConfig request to add master 127.0.71.123:42445: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "4316b0d33e304556b18eb80895390ee6"
member_type: NON_VOTER
last_known_addr {
host: "127.0.71.123"
port: 42445
}
attrs {
promote: true
}
}
dest_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b"
cas_config_opid_index: -1
I20260929 01:59:42.094246 11092 consensus_queue.cc:237] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: true } }
I20260929 01:59:42.097432 11026 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [term 1 FOLLOWER]: Refusing update from remote peer c40cd5c3ac114930a92ebe1d974dbe1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20260929 01:59:42.098335 11079 consensus_peers.cc:597] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b -> Peer 4316b0d33e304556b18eb80895390ee6 (127.0.71.123:42445): Couldn't send request to peer 4316b0d33e304556b18eb80895390ee6. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260929 01:59:42.098642 11514 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:59:42.098719 11180 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [term 1 FOLLOWER]: Refusing update from remote peer c40cd5c3ac114930a92ebe1d974dbe1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 01:59:42.099259 11267 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 01:59:42.101917 11267 raft_consensus.cc:2955] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4316b0d33e304556b18eb80895390ee6 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: true } } }
I20260929 01:59:42.102599 11026 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4316b0d33e304556b18eb80895390ee6 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: true } } }
I20260929 01:59:42.103245 11180 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 4316b0d33e304556b18eb80895390ee6 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: true } } }
I20260929 01:59:42.106608 11267 catalog_manager.cc:7355] Successfully completed master ChangeConfig request to add master 127.0.71.123:42445
I20260929 01:59:42.107321 11414 master.cc:562] Master@127.0.71.123:42445 shutting down...
I20260929 01:59:42.123212 11517 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: true } } }
I20260929 01:59:42.123961 11517 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:42.128262 11514 sys_catalog.cc:455] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: true } } }
I20260929 01:59:42.128652 11514 sys_catalog.cc:458] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: This master's current role is: LEADER
I20260929 01:59:42.133687 11519 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: true } } }
I20260929 01:59:42.134927 11519 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:42.137426 11518 sys_catalog.cc:455] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: true } } }
I20260929 01:59:42.137969 11518 sys_catalog.cc:458] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: This master's current role is: LEADER
I20260929 01:59:42.138422 11518 sys_catalog.cc:455] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: true } } }
I20260929 01:59:42.138723 11518 sys_catalog.cc:458] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: This master's current role is: LEADER
I20260929 01:59:42.165155 11414 master.cc:584] Master@127.0.71.123:42445 shutdown complete.
I20260929 01:59:42.165266 11414 master_runner.cc:306] Clearing existing system tablet
I20260929 01:59:42.165728 11414 fs_manager.cc:362] Metadata directory not provided
I20260929 01:59:42.165838 11414 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:59:42.168982 11414 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.003s sys 0.001s
I20260929 01:59:42.170753 11524 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:42.171185 11414 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:59:42.171463 11414 fs_manager.cc:730] Time spent opening block manager: real 0.002s user 0.001s sys 0.001s
I20260929 01:59:42.171553 11414 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "4316b0d33e304556b18eb80895390ee6"
format_stamp: "Formatted at 2026-09-29 01:59:41 on dist-test-slave-vz23"
I20260929 01:59:42.176174 11414 ts_tablet_manager.cc:1962] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 01:59:42.183454 11414 ts_tablet_manager.cc:1975] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 01:59:42.183629 11414 log.cc:1199] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260929 01:59:42.184039 11414 ts_tablet_manager.cc:1996] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6: Deleting consensus metadata
I20260929 01:59:42.184437 11414 master_runner.cc:316] Copying system tablet from 127.0.71.125:36257
I20260929 01:59:42.187613 11414 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6: tablet copy: Beginning tablet copy session from remote peer at address 127.0.71.125:36257
I20260929 01:59:42.201326 11113 tablet_copy_service.cc:140] P c40cd5c3ac114930a92ebe1d974dbe1b: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 4316b0d33e304556b18eb80895390ee6 ({username='kudu', principal='kudu/127.0.71.123@KRBTEST.COM'} at 127.0.0.1:59678)
I20260929 01:59:42.201639 11113 tablet_copy_service.cc:161] P c40cd5c3ac114930a92ebe1d974dbe1b: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 4316b0d33e304556b18eb80895390ee6 at {username='kudu', principal='kudu/127.0.71.123@KRBTEST.COM'} at 127.0.0.1:59678: session id = 4316b0d33e304556b18eb80895390ee6-00000000000000000000000000000000
I20260929 01:59:42.204864 11113 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 01:59:42.206982 11414 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 01:59:42.214248 11414 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6: tablet copy: Starting download of 0 data blocks...
I20260929 01:59:42.214561 11414 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6: tablet copy: Starting download of 1 WAL segments...
I20260929 01:59:42.217320 11414 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 01:59:42.220846 11113 tablet_copy_service.cc:342] P c40cd5c3ac114930a92ebe1d974dbe1b: Request end of tablet copy session 4316b0d33e304556b18eb80895390ee6-00000000000000000000000000000000 received from {username='kudu', principal='kudu/127.0.71.123@KRBTEST.COM'} at 127.0.0.1:59678
I20260929 01:59:42.221072 11113 tablet_copy_service.cc:434] P c40cd5c3ac114930a92ebe1d974dbe1b: ending tablet copy session 4316b0d33e304556b18eb80895390ee6-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 4316b0d33e304556b18eb80895390ee6
W20260929 01:59:42.228403 11414 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260929 01:59:42.231647 11414 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:59:42.234305 11531 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:42.235877 11532 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:59:42.236534 11414 server_base.cc:1061] running on GCE node
W20260929 01:59:42.237036 11534 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:59:42.237430 11414 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:59:42.238201 11414 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:59:42.239405 11414 hybrid_clock.cc:648] HybridClock initialized: now 1790647182239361 us; error 35 us; skew 500 ppm
I20260929 01:59:42.239751 11414 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:59:42 dist-test-slave-vz23 krb5kdc[10688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:59:42.241848 11414 init.cc:377] Logged in from keytab as kudu/127.0.71.123@KRBTEST.COM (short username kudu)
I20260929 01:59:42.249183 11414 webserver.cc:533] Webserver started at http://127.0.71.123:44523/ using document root <none> and password file <none>
I20260929 01:59:42.249488 11414 fs_manager.cc:362] Metadata directory not provided
I20260929 01:59:42.249601 11414 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:59:42.251987 11414 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.000s
I20260929 01:59:42.254000 11541 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:42.254447 11414 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.000s
I20260929 01:59:42.254604 11414 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "4316b0d33e304556b18eb80895390ee6"
format_stamp: "Formatted at 2026-09-29 01:59:41 on dist-test-slave-vz23"
I20260929 01:59:42.254807 11414 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:59:42.289436 11414 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:59:42.296509 11414 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:59:42.320353 11414 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.123:42445
I20260929 01:59:42.320380 11592 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.123:42445 every 8 connection(s)
I20260929 01:59:42.321698 11414 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
I20260929 01:59:42.325785 11593 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:59:42.328198 11593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6: Bootstrap starting.
I20260929 01:59:42.340695 11593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:59:42.341112 11593 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6: Bootstrap complete.
I20260929 01:59:42.342053 11593 raft_consensus.cc:359] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: true } }
I20260929 01:59:42.342283 11593 raft_consensus.cc:740] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 4316b0d33e304556b18eb80895390ee6, State: Initialized, Role: LEARNER
I20260929 01:59:42.342521 11593 consensus_queue.cc:260] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: true } }
I20260929 01:59:42.343401 11593 sys_catalog.cc:565] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:59:42.343302 11596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: true } } }
I20260929 01:59:42.343611 11596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6 [sys.catalog]: This master's current role is: LEARNER
I20260929 01:59:42.351969 11607 catalog_manager.cc:1308] Loaded cluster ID: 976be3b2fe9c4e609f2f1aaf1e0dc177
I20260929 01:59:42.352113 11607 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6: loading cluster ID for follower catalog manager: success
I20260929 01:59:42.355916 11607 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6: acquiring CA information for follower catalog manager: success
I20260929 01:59:42.358402 11402 heartbeater.cc:499] Master 127.0.71.125:36257 was elected leader, sending a full tablet report...
I20260929 01:59:42.360347 11607 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 01:59:42.375011 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 11272
Sep 29 01:59:42 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 01:59:42.391848 11395 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:59:42.484139 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 11272
I20260929 01:59:42.484488 11568 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260929 01:59:42.485628 11596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c40cd5c3ac114930a92ebe1d974dbe1b. Latest consensus state: current_term: 1 leader_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: true } } }
I20260929 01:59:42.485879 11596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6 [sys.catalog]: This master's current role is: LEARNER
I20260929 01:59:42.508123 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:38977
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:38413,127.0.71.125:36257,127.0.71.124:36677
--webserver_port=32989
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:44609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:38413,127.0.71.125:36257,127.0.71.124:36677,127.0.71.123:42445 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:59:42.850960 11611 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:59:42.851270 11611 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:59:42.851364 11611 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:59:42.860131 11611 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:59:42.860318 11611 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:59:42.871287 11611 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:44609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:38977
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=32989
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:38413,127.0.71.125:36257,127.0.71.124:36677,127.0.71.123:42445
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:59:42.872933 11611 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:59:42.874313 11611 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:59:42.884931 11617 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:42.884982 11616 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:42.885862 11619 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:59:42.886703 11611 server_base.cc:1061] running on GCE node
I20260929 01:59:42.887286 11611 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:59:42.888259 11611 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:59:42.889503 11611 hybrid_clock.cc:648] HybridClock initialized: now 1790647182889446 us; error 83 us; skew 500 ppm
I20260929 01:59:42.889876 11611 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:59:42 dist-test-slave-vz23 krb5kdc[10688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:59:42.893381 11611 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:59:42.895253 11611 webserver.cc:533] Webserver started at http://127.0.71.65:32989/ using document root <none> and password file <none>
I20260929 01:59:42.895967 11611 fs_manager.cc:362] Metadata directory not provided
I20260929 01:59:42.896116 11611 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:59:42.901707 11611 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 01:59:42.904556 11626 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:42.905965 11611 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:59:42.906144 11611 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "4a016aaa46334430a446bf502f0a94ff"
format_stamp: "Formatted at 2026-09-29 01:59:41 on dist-test-slave-vz23"
I20260929 01:59:42.906876 11611 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:59:42.925485 11611 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:59:42.929374 11611 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:59:42.929642 11611 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:59:42.930560 11611 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:59:42.932112 11611 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:59:42.932237 11611 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:42.932367 11611 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:59:42.932472 11611 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:42.961472 11514 sys_catalog.cc:455] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: true } } }
I20260929 01:59:42.961930 11514 sys_catalog.cc:458] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: This master's current role is: LEADER
I20260929 01:59:42.962534 11514 raft_consensus.cc:1064] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b: attempting to promote NON_VOTER 4316b0d33e304556b18eb80895390ee6 to VOTER
I20260929 01:59:42.963527 11514 consensus_queue.cc:237] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: false } }
I20260929 01:59:42.966090 11180 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [term 1 FOLLOWER]: Refusing update from remote peer c40cd5c3ac114930a92ebe1d974dbe1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:59:42.966172 11568 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6 [term 1 LEARNER]: Refusing update from remote peer c40cd5c3ac114930a92ebe1d974dbe1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:59:42.966890 11697 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:59:42.967382 11699 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:59:42.969208 11026 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [term 1 FOLLOWER]: Refusing update from remote peer c40cd5c3ac114930a92ebe1d974dbe1b: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 01:59:42.970043 11697 consensus_queue.cc:1048] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [LEADER]: Connected to new peer: Peer: permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 01:59:42.973420 11699 raft_consensus.cc:2955] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4316b0d33e304556b18eb80895390ee6 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: false } } }
I20260929 01:59:42.974296 11026 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4316b0d33e304556b18eb80895390ee6 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: false } } }
I20260929 01:59:42.979910 11180 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4316b0d33e304556b18eb80895390ee6 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: false } } }
I20260929 01:59:42.980931 11567 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 4316b0d33e304556b18eb80895390ee6 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: false } } }
I20260929 01:59:42.983317 11724 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: false } } }
I20260929 01:59:42.984215 11724 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:42.986521 11596 sys_catalog.cc:455] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: false } } }
I20260929 01:59:42.986830 11596 sys_catalog.cc:458] T 00000000000000000000000000000000 P 4316b0d33e304556b18eb80895390ee6 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:42.988222 11697 sys_catalog.cc:455] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: false } } }
I20260929 01:59:42.988583 11697 sys_catalog.cc:458] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: This master's current role is: LEADER
I20260929 01:59:42.988265 11514 sys_catalog.cc:455] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: false } } }
I20260929 01:59:42.988816 11514 sys_catalog.cc:458] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: This master's current role is: LEADER
I20260929 01:59:42.989114 11514 sys_catalog.cc:455] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: false } } }
I20260929 01:59:42.989360 11514 sys_catalog.cc:458] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: This master's current role is: LEADER
I20260929 01:59:42.989665 11514 sys_catalog.cc:455] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: false } } }
I20260929 01:59:42.989917 11514 sys_catalog.cc:458] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: This master's current role is: LEADER
I20260929 01:59:42.990942 11726 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: false } } }
I20260929 01:59:42.991184 11726 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7878d5fe172a493a8f052281b39a28b7 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:42.998857 11611 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:38977
I20260929 01:59:42.998937 11745 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:38977 every 8 connection(s)
I20260929 01:59:43.000346 11611 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:59:43.007266 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 11611
Sep 29 01:59:43 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 01:59:43 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
Sep 29 01:59:43 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 01:59:43 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647182, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:59:43.047654 11747 heartbeater.cc:344] Connected to a master server at 127.0.71.126:38413
I20260929 01:59:43.047684 11746 heartbeater.cc:344] Connected to a master server at 127.0.71.124:36677
I20260929 01:59:43.048118 11746 heartbeater.cc:461] Registering TS with master...
I20260929 01:59:43.048230 11749 heartbeater.cc:344] Connected to a master server at 127.0.71.125:36257
I20260929 01:59:43.048347 11749 heartbeater.cc:461] Registering TS with master...
I20260929 01:59:43.048462 11748 heartbeater.cc:344] Connected to a master server at 127.0.71.123:42445
I20260929 01:59:43.048113 11747 heartbeater.cc:461] Registering TS with master...
I20260929 01:59:43.049011 11748 heartbeater.cc:461] Registering TS with master...
I20260929 01:59:43.050024 11748 heartbeater.cc:507] Master 127.0.71.123:42445 requested a full tablet report, sending...
I20260929 01:59:43.050073 11749 heartbeater.cc:507] Master 127.0.71.125:36257 requested a full tablet report, sending...
I20260929 01:59:43.050164 11747 heartbeater.cc:507] Master 127.0.71.126:38413 requested a full tablet report, sending...
I20260929 01:59:43.050889 11746 heartbeater.cc:507] Master 127.0.71.124:36677 requested a full tablet report, sending...
I20260929 01:59:43.052242 11092 ts_manager.cc:194] Re-registered known tserver with Master: 4a016aaa46334430a446bf502f0a94ff (127.0.71.65:38977)
I20260929 01:59:43.052284 11169 ts_manager.cc:194] Re-registered known tserver with Master: 4a016aaa46334430a446bf502f0a94ff (127.0.71.65:38977)
I20260929 01:59:43.052378 11017 ts_manager.cc:194] Re-registered known tserver with Master: 4a016aaa46334430a446bf502f0a94ff (127.0.71.65:38977)
I20260929 01:59:43.053252 11558 ts_manager.cc:194] Registered new tserver with Master: 4a016aaa46334430a446bf502f0a94ff (127.0.71.65:38977)
I20260929 01:59:43.053390 11092 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:47881
I20260929 01:59:43.054747 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:59:43.064358 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 01:59:43.064554 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 10985
I20260929 01:59:43.065446 11047 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:59:43.178627 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 10985
I20260929 01:59:43.200897 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:38413
--webserver_interface=127.0.71.126
--webserver_port=39281
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:44609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:38413,127.0.71.125:36257,127.0.71.124:36677
--master_addresses=127.0.71.126:38413,127.0.71.125:36257,127.0.71.124:36677,127.0.71.123:42445 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:59:43.503352 11077 consensus_peers.cc:597] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b -> Peer 93831b88e43f471dbf8af58a90aca3c9 (127.0.71.126:38413): Couldn't send request to peer 93831b88e43f471dbf8af58a90aca3c9. Status: Network error: Client connection negotiation failed: client connection to 127.0.71.126:38413: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260929 01:59:43.557549 11760 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:59:43.557885 11760 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:59:43.557989 11760 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:59:43.567216 11760 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 01:59:43.567324 11760 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:59:43.567392 11760 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 01:59:43.567445 11760 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 01:59:43.579061 11760 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:44609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:38413,127.0.71.125:36257,127.0.71.124:36677,127.0.71.123:42445
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:38413
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=39281
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:59:43.580561 11760 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:59:43.581809 11760 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:59:43.589856 11768 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:43.589903 11767 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:43.590991 11770 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:59:43.591466 11760 server_base.cc:1061] running on GCE node
I20260929 01:59:43.592139 11760 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:59:43.593539 11760 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:59:43.594738 11760 hybrid_clock.cc:648] HybridClock initialized: now 1790647183594689 us; error 45 us; skew 500 ppm
I20260929 01:59:43.595079 11760 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:59:43 dist-test-slave-vz23 krb5kdc[10688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:59:43.598378 11760 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 01:59:43.600050 11760 webserver.cc:533] Webserver started at http://127.0.71.126:39281/ using document root <none> and password file <none>
I20260929 01:59:43.600787 11760 fs_manager.cc:362] Metadata directory not provided
I20260929 01:59:43.600898 11760 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:59:43.606024 11760 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.004s sys 0.000s
I20260929 01:59:43.608824 11777 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:43.610064 11760 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 01:59:43.610246 11760 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "93831b88e43f471dbf8af58a90aca3c9"
format_stamp: "Formatted at 2026-09-29 01:59:38 on dist-test-slave-vz23"
I20260929 01:59:43.611006 11760 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:59:43.648656 11760 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:59:43.660732 11760 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:59:43.661044 11760 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:59:43.682148 11760 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:38413
I20260929 01:59:43.682190 11828 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:38413 every 8 connection(s)
I20260929 01:59:43.683810 11760 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 01:59:43.687666 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 11760
I20260929 01:59:43.687960 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 11611
I20260929 01:59:43.691357 11829 sys_catalog.cc:263] Verifying existing consensus state
I20260929 01:59:43.695659 11829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9: Bootstrap starting.
I20260929 01:59:43.705751 11741 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:59:43.716747 11829 log.cc:826] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9: Log is configured to *not* fsync() on all Append() calls
I20260929 01:59:43.726105 11829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9: Bootstrap replayed 1/1 log segments. Stats: ops{read=6 overwritten=0 applied=6 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 01:59:43.726675 11829 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9: Bootstrap complete.
I20260929 01:59:43.733676 11829 raft_consensus.cc:359] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: false } }
I20260929 01:59:43.734390 11829 raft_consensus.cc:740] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: 93831b88e43f471dbf8af58a90aca3c9, State: Initialized, Role: FOLLOWER
I20260929 01:59:43.735195 11829 consensus_queue.cc:260] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 6, Last appended: 1.6, Last appended by leader: 6, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: false } }
I20260929 01:59:43.736071 11835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: false } } }
I20260929 01:59:43.736414 11835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 01:59:43.737162 11829 sys_catalog.cc:565] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 01:59:43.753652 11846 catalog_manager.cc:1308] Loaded cluster ID: 976be3b2fe9c4e609f2f1aaf1e0dc177
I20260929 01:59:43.755568 11846 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9: loading cluster ID for follower catalog manager: success
I20260929 01:59:43.761114 11846 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9: acquiring CA information for follower catalog manager: success
I20260929 01:59:43.764007 11846 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Sep 29 01:59:43 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647183, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 01:59:43.836859 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 11611
I20260929 01:59:43.860414 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:38977
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:38413,127.0.71.125:36257,127.0.71.124:36677
--webserver_port=32989
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:44609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:38413,127.0.71.125:36257,127.0.71.124:36677,127.0.71.123:42445 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 01:59:43.921126 11699 sys_catalog.cc:455] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: false } } }
I20260929 01:59:43.921402 11699 sys_catalog.cc:458] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: This master's current role is: LEADER
I20260929 01:59:43.931829 11835 sys_catalog.cc:455] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c40cd5c3ac114930a92ebe1d974dbe1b. Latest consensus state: current_term: 1 leader_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: false } } }
I20260929 01:59:43.932111 11835 sys_catalog.cc:458] T 00000000000000000000000000000000 P 93831b88e43f471dbf8af58a90aca3c9 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 01:59:44.217806 11853 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:59:44.218145 11853 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:59:44.218298 11853 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:59:44.226850 11853 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:59:44.227082 11853 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:59:44.238670 11853 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:44609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:38977
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=32989
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:38413,127.0.71.125:36257,127.0.71.124:36677,127.0.71.123:42445
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:59:44.240327 11853 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:59:44.241672 11853 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:59:44.251061 11861 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:44.251063 11862 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:44.251822 11864 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:59:44.252511 11853 server_base.cc:1061] running on GCE node
I20260929 01:59:44.253084 11853 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:59:44.254084 11853 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:59:44.255319 11853 hybrid_clock.cc:648] HybridClock initialized: now 1790647184255284 us; error 58 us; skew 500 ppm
I20260929 01:59:44.255632 11853 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:59:44 dist-test-slave-vz23 krb5kdc[10688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:59:44.259085 11853 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:59:44.260882 11853 webserver.cc:533] Webserver started at http://127.0.71.65:32989/ using document root <none> and password file <none>
I20260929 01:59:44.261602 11853 fs_manager.cc:362] Metadata directory not provided
I20260929 01:59:44.261751 11853 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:59:44.267627 11853 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.000s sys 0.006s
I20260929 01:59:44.270560 11871 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:44.272001 11853 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 01:59:44.272198 11853 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "4a016aaa46334430a446bf502f0a94ff"
format_stamp: "Formatted at 2026-09-29 01:59:41 on dist-test-slave-vz23"
I20260929 01:59:44.272987 11853 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:59:44.312413 11853 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:59:44.316629 11853 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:59:44.316907 11853 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:59:44.317867 11853 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:59:44.319540 11853 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:59:44.319638 11853 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:44.319761 11853 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:59:44.319861 11853 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:44.325683 11855 sys_catalog.cc:455] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "93831b88e43f471dbf8af58a90aca3c9" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38413 } } peers { permanent_uuid: "c40cd5c3ac114930a92ebe1d974dbe1b" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36257 } } peers { permanent_uuid: "7878d5fe172a493a8f052281b39a28b7" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 36677 } } peers { permanent_uuid: "4316b0d33e304556b18eb80895390ee6" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 42445 } attrs { promote: false } } }
I20260929 01:59:44.325970 11855 sys_catalog.cc:458] T 00000000000000000000000000000000 P c40cd5c3ac114930a92ebe1d974dbe1b [sys.catalog]: This master's current role is: LEADER
I20260929 01:59:44.360733 11853 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:38977
I20260929 01:59:44.360821 11983 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:38977 every 8 connection(s)
I20260929 01:59:44.362411 11853 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:59:44.364771 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 11853
Sep 29 01:59:44 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647159, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for hive/127.0.0.1@KRBTEST.COM
Sep 29 01:59:44 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
Sep 29 01:59:44 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 01:59:44.408190 11986 heartbeater.cc:344] Connected to a master server at 127.0.71.123:42445
I20260929 01:59:44.408707 11986 heartbeater.cc:461] Registering TS with master...
I20260929 01:59:44.409885 11986 heartbeater.cc:507] Master 127.0.71.123:42445 requested a full tablet report, sending...
Sep 29 01:59:44 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 01:59:44 dist-test-slave-vz23 krb5kdc[10688](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 01:59:44.412120 11558 ts_manager.cc:194] Re-registered known tserver with Master: 4a016aaa46334430a446bf502f0a94ff (127.0.71.65:38977)
I20260929 01:59:44.415917 11987 heartbeater.cc:344] Connected to a master server at 127.0.71.125:36257
I20260929 01:59:44.416065 11987 heartbeater.cc:461] Registering TS with master...
I20260929 01:59:44.416632 11987 heartbeater.cc:507] Master 127.0.71.125:36257 requested a full tablet report, sending...
I20260929 01:59:44.417729 11092 ts_manager.cc:194] Re-registered known tserver with Master: 4a016aaa46334430a446bf502f0a94ff (127.0.71.65:38977)
I20260929 01:59:44.418952 11092 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:60789
I20260929 01:59:44.422696 11984 heartbeater.cc:344] Connected to a master server at 127.0.71.124:36677
I20260929 01:59:44.422840 11984 heartbeater.cc:461] Registering TS with master...
I20260929 01:59:44.423143 11985 heartbeater.cc:344] Connected to a master server at 127.0.71.126:38413
I20260929 01:59:44.423254 11984 heartbeater.cc:507] Master 127.0.71.124:36677 requested a full tablet report, sending...
I20260929 01:59:44.423313 11985 heartbeater.cc:461] Registering TS with master...
I20260929 01:59:44.423633 11985 heartbeater.cc:507] Master 127.0.71.126:38413 requested a full tablet report, sending...
I20260929 01:59:44.424588 11169 ts_manager.cc:194] Re-registered known tserver with Master: 4a016aaa46334430a446bf502f0a94ff (127.0.71.65:38977)
I20260929 01:59:44.425436 11794 ts_manager.cc:194] Registered new tserver with Master: 4a016aaa46334430a446bf502f0a94ff (127.0.71.65:38977)
I20260929 01:59:44.526444 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 11853
W20260929 01:59:44.527745 12001 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:59:44.523966 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:39680 (local address 127.0.71.65:38977)
0929 01:59:44.524696 (+ 730us) server_negotiation.cc:207] Beginning negotiation
0929 01:59:44.524707 (+ 11us) server_negotiation.cc:400] Waiting for connection header
0929 01:59:44.524742 (+ 35us) server_negotiation.cc:408] Connection header received
0929 01:59:44.524819 (+ 77us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:59:44.524828 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:59:44.525132 (+ 304us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:59:44.525340 (+ 208us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:59:44.526355 (+ 1015us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:39680: BlockingRecv error: recv got EOF from 127.0.0.1:39680 (error 108)
Metrics: {"server-negotiator.queue_time_us":570,"thread_start_us":350,"threads_started":1}
W20260929 01:59:44.529055 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 11853: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:38977: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:59:44.529177 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 11853
I20260929 01:59:44.547483 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:38977
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:38413,127.0.71.125:36257,127.0.71.124:36677
--webserver_port=32989
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:44609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:38413,127.0.71.125:36257,127.0.71.124:36677,127.0.71.123:42445
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:59:44.889032 12003 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:59:44.889372 12003 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:59:44.889536 12003 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 01:59:44.889635 12003 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:59:44.897962 12003 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:59:44.898222 12003 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:59:44.909466 12003 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:44609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:38977
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=32989
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:38413,127.0.71.125:36257,127.0.71.124:36677,127.0.71.123:42445
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:59:44.911104 12003 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:59:44.912521 12003 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:59:44.922150 12008 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:44.922209 12009 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:44.923362 12011 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:59:44.923519 12003 server_base.cc:1061] running on GCE node
I20260929 01:59:44.924109 12003 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:59:44.925290 12003 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:59:44.926558 12003 hybrid_clock.cc:648] HybridClock initialized: now 1790647184926474 us; error 62 us; skew 500 ppm
I20260929 01:59:44.926954 12003 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:59:44 dist-test-slave-vz23 krb5kdc[10688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647184, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:59:44.930354 12003 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:59:44.932029 12003 webserver.cc:533] Webserver started at http://127.0.71.65:32989/ using document root <none> and password file <none>
I20260929 01:59:44.932830 12003 fs_manager.cc:362] Metadata directory not provided
I20260929 01:59:44.932912 12003 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:59:44.938938 12003 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.002s
I20260929 01:59:44.942067 12018 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:44.943545 12003 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 01:59:44.943750 12003 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "4a016aaa46334430a446bf502f0a94ff"
format_stamp: "Formatted at 2026-09-29 01:59:41 on dist-test-slave-vz23"
I20260929 01:59:44.944600 12003 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:59:44.969537 12003 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:59:44.973636 12003 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:59:44.973944 12003 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:59:44.974936 12003 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:59:44.976562 12003 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:59:44.976667 12003 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:44.976820 12003 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:59:44.976892 12003 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 01:59:44.977301 12003 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 01:59:44.977386 12003 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 01:59:44.977476 12003 fault_injection.cc:59] injecting fault for kudu (pid 12003): FLAGS_fault_before_start (process will exit)
I20260929 01:59:44.997191 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:38977
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:38413,127.0.71.125:36257,127.0.71.124:36677
--webserver_port=32989
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:44609
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:38413,127.0.71.125:36257,127.0.71.124:36677,127.0.71.123:42445 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 01:59:45.361590 12026 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 01:59:45.361925 12026 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 01:59:45.362129 12026 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 01:59:45.370680 12026 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 01:59:45.370939 12026 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 01:59:45.381839 12026 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:44609
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:38977
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=32989
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:38413,127.0.71.125:36257,127.0.71.124:36677,127.0.71.123:42445
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 01:59:45.383499 12026 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 01:59:45.384938 12026 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 01:59:45.394726 12033 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:45.394754 12034 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 01:59:45.395306 12036 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 01:59:45.396111 12026 server_base.cc:1061] running on GCE node
I20260929 01:59:45.396762 12026 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 01:59:45.397758 12026 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 01:59:45.398983 12026 hybrid_clock.cc:648] HybridClock initialized: now 1790647185398948 us; error 63 us; skew 500 ppm
I20260929 01:59:45.399317 12026 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 01:59:45 dist-test-slave-vz23 krb5kdc[10688](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647185, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 01:59:45.402629 12026 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 01:59:45.404201 12026 webserver.cc:533] Webserver started at http://127.0.71.65:32989/ using document root <none> and password file <none>
I20260929 01:59:45.404973 12026 fs_manager.cc:362] Metadata directory not provided
I20260929 01:59:45.405145 12026 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 01:59:45.410507 12026 fs_manager.cc:714] Time spent opening directory manager: real 0.003s user 0.005s sys 0.000s
I20260929 01:59:45.413676 12043 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:45.415342 12026 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.003s sys 0.000s
I20260929 01:59:45.415526 12026 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "4a016aaa46334430a446bf502f0a94ff"
format_stamp: "Formatted at 2026-09-29 01:59:41 on dist-test-slave-vz23"
I20260929 01:59:45.416354 12026 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 01:59:45.440651 12026 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 01:59:45.444746 12026 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 01:59:45.445096 12026 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 01:59:45.446113 12026 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 01:59:45.447855 12026 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 01:59:45.447949 12026 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:45.448100 12026 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 01:59:45.448205 12026 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 01:59:45.485138 12026 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:38977
I20260929 01:59:45.485229 12155 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:38977 every 8 connection(s)
I20260929 01:59:45.486688 12026 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 01:59:45.488754 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 12026
I20260929 01:59:45.489039 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 12026
W20260929 01:59:45.538645 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 12026: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:38977: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:59:45.538789 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 12026
I20260929 01:59:45.555478 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 11760
W20260929 01:59:45.556239 12028 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:59:45.506262 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:52583 (local address 127.0.71.126:38413)
0929 01:59:45.506495 (+ 233us) server_negotiation.cc:207] Beginning negotiation
0929 01:59:45.506503 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0929 01:59:45.535037 (+ 28534us) server_negotiation.cc:408] Connection header received
0929 01:59:45.536330 (+ 1293us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:59:45.536337 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:59:45.536431 (+ 94us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:59:45.536591 (+ 160us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:59:45.555303 (+ 18712us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:52583: BlockingRecv error: recv got EOF from 127.0.71.65:52583 (error 108)
Metrics: {"server-negotiator.queue_time_us":79}
W20260929 01:59:45.556267 12163 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:59:45.491433 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:47739 (local address 127.0.71.123:42445)
0929 01:59:45.527808 (+ 36375us) server_negotiation.cc:207] Beginning negotiation
0929 01:59:45.527818 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0929 01:59:45.527849 (+ 31us) server_negotiation.cc:408] Connection header received
0929 01:59:45.536256 (+ 8407us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:59:45.536265 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:59:45.536358 (+ 93us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:59:45.536541 (+ 183us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:59:45.555383 (+ 18842us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:47739: BlockingRecv error: recv got EOF from 127.0.71.65:47739 (error 108)
Metrics: {"server-negotiator.queue_time_us":36182,"thread_start_us":36436,"threads_started":1}
W20260929 01:59:45.556267 12168 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:59:45.527746 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:52251 (local address 127.0.71.125:36257)
0929 01:59:45.533503 (+ 5757us) server_negotiation.cc:207] Beginning negotiation
0929 01:59:45.533512 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0929 01:59:45.535578 (+ 2066us) server_negotiation.cc:408] Connection header received
0929 01:59:45.537070 (+ 1492us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:59:45.537077 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:59:45.537168 (+ 91us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:59:45.537290 (+ 122us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:59:45.555291 (+ 18001us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:52251: BlockingRecv error: recv got EOF from 127.0.71.65:52251 (error 108)
Metrics: {"server-negotiator.queue_time_us":5028,"thread_start_us":4912,"threads_started":1}
W20260929 01:59:45.557602 12161 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:59:45.490227 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:55159 (local address 127.0.71.124:36677)
0929 01:59:45.530700 (+ 40473us) server_negotiation.cc:207] Beginning negotiation
0929 01:59:45.530710 (+ 10us) server_negotiation.cc:400] Waiting for connection header
0929 01:59:45.530740 (+ 30us) server_negotiation.cc:408] Connection header received
0929 01:59:45.536928 (+ 6188us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:59:45.536937 (+ 9us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:59:45.537046 (+ 109us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:59:45.537157 (+ 111us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:59:45.556828 (+ 19671us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:55159: BlockingRecv error: recv got EOF from 127.0.71.65:55159 (error 108)
Metrics: {"server-negotiator.queue_time_us":40315,"thread_start_us":40533,"threads_started":1}
W20260929 01:59:45.558467 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 11760: Not authorized: Client connection negotiation failed: client connection to 127.0.71.126:38413: server requires authentication, but client does not have Kerberos credentials available
I20260929 01:59:45.558609 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 11760
W20260929 01:59:45.558650 12028 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 01:59:45.556725 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:56924 (local address 127.0.71.126:38413)
0929 01:59:45.556926 (+ 201us) server_negotiation.cc:207] Beginning negotiation
0929 01:59:45.556933 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0929 01:59:45.557254 (+ 321us) server_negotiation.cc:408] Connection header received
0929 01:59:45.557409 (+ 155us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 01:59:45.557414 (+ 5us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 01:59:45.557503 (+ 89us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 01:59:45.557638 (+ 135us) server_negotiation.cc:227] Negotiated authn=SASL
0929 01:59:45.558378 (+ 740us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:56924: BlockingRecv error: recv got EOF from 127.0.0.1:56924 (error 108)
Metrics: {"server-negotiator.queue_time_us":76}
I20260929 01:59:45.578662 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 11061
I20260929 01:59:45.579885 11123 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:59:45.704282 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 11061
I20260929 01:59:45.727061 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 11138
I20260929 01:59:45.728346 11200 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:59:45.843467 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 11138
I20260929 01:59:45.864987 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 11414
I20260929 01:59:45.866170 11588 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 01:59:45.983289 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 11414
Shutting down hive metastore.
2026-09-29T01:59:46Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestAddMaster/10 (29234 ms)
[ RUN ] ExternalMiniClusterTest.TestAddMaster/11
Loading random data
Initializing database '/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 01:59:46 dist-test-slave-vz23 krb5kdc[12179](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 01:59:46 dist-test-slave-vz23 krb5kdc[12179](info): set up 2 sockets
Sep 29 01:59:46 dist-test-slave-vz23 krb5kdc[12179](info): commencing operation
krb5kdc: starting...
W20260929 01:59:48.524857 285 mini_kdc.cc:121] Time spent starting KDC: real 2.112s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 01:59:48 dist-test-slave-vz23 krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T01:59:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:59:48Z Disabled control of system clock
2026-09-29T01:59:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:59:48Z Disabled control of system clock
2026-09-29T01:59:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:59:48Z Disabled control of system clock
2026-09-29T01:59:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:59:48Z Disabled control of system clock
2026-09-29T01:59:48Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T01:59:48Z Disabled control of system clock
WARNING: no policy specified for hive/127.0.0.1@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "hive/127.0.0.1@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/hive_127.0.0.1.keytab.
Entry for principal hive/127.0.0.1 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/hive_127.0.0.1.keytab.
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Metastore connection URL: jdbc:derby:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/metadb;create=true
Metastore Connection Driver : org.apache.derby.jdbc.EmbeddedDriver
Metastore connection User: APP
Starting metastore schema initialization to 3.1.0
Initialization script hive-schema-3.1.0.derby.sql
Initialization script completed
schemaTool completed
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/krb5.conf
2026-09-29 01:59:56: Starting Hive Metastore Server
Picked up JAVA_TOOL_OPTIONS: -Djava.net.preferIPv4Stack=true -Dderby.locks.deadlockTimeout=1 -Dderby.stream.error.file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/derby.log -Djava.security.properties=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/security.properties -Djava.security.krb5.conf=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/krb5.conf
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hive/lib/log4j-slf4j-impl-2.10.0.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:/tmp/dist-test-taskniqJIA/thirdparty/installed/common/opt/hadoop/share/hadoop/common/lib/slf4j-reload4j-1.7.36.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
Starting hive metastore on port 0
Sep 29 01:59:58 dist-test-slave-vz23 krb5kdc[12179](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790647198, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 02:00:04 dist-test-slave-vz23 krb5kdc[12179](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790647204, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Sep 29 02:00:04 dist-test-slave-vz23 krb5kdc[12179](info): AS_REQ (1 etypes {17}) 127.0.0.1: ISSUE: authtime 1790647204, etypes {rep=17 tkt=17 ses=17}, hive/127.0.0.1@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:05.276253 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:42039
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:36581,127.0.71.85:45153,127.0.71.86:42537,127.0.71.87:34613,127.0.71.88:45991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:42039,127.0.71.125:40905,127.0.71.124:33787 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:05.618036 12483 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:05.618419 12483 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:05.618571 12483 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:05.627138 12483 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:00:05.627298 12483 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:05.627418 12483 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:00:05.627514 12483 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:00:05.638402 12483 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:36581,127.0.71.85:45153,127.0.71.86:42537,127.0.71.87:34613,127.0.71.88:45991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:42039,127.0.71.125:40905,127.0.71.124:33787
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:42039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:05.640033 12483 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:05.641347 12483 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:05.649689 12489 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:05.649660 12488 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:05.650118 12491 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:05.650890 12483 server_base.cc:1061] running on GCE node
I20260929 02:00:05.651716 12483 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:05.653378 12483 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:05.654605 12483 hybrid_clock.cc:648] HybridClock initialized: now 1790647205654474 us; error 102 us; skew 500 ppm
I20260929 02:00:05.654968 12483 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:05 dist-test-slave-vz23 krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:05.658202 12483 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 02:00:05.659853 12483 webserver.cc:533] Webserver started at http://127.0.71.126:42021/ using document root <none> and password file <none>
I20260929 02:00:05.660621 12483 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:05.660754 12483 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:05.661088 12483 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:05.663491 12483 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "b97759321664486dad92e9d63f3f5332"
format_stamp: "Formatted at 2026-09-29 02:00:05 on dist-test-slave-vz23"
I20260929 02:00:05.664247 12483 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "b97759321664486dad92e9d63f3f5332"
format_stamp: "Formatted at 2026-09-29 02:00:05 on dist-test-slave-vz23"
I20260929 02:00:05.669462 12483 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 02:00:05.672853 12498 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:05.674196 12483 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:00:05.674413 12483 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "b97759321664486dad92e9d63f3f5332"
format_stamp: "Formatted at 2026-09-29 02:00:05 on dist-test-slave-vz23"
I20260929 02:00:05.674612 12483 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:05.695247 12483 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:05.705539 12483 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:05.705874 12483 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:05.727979 12483 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:42039
I20260929 02:00:05.728021 12549 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:42039 every 8 connection(s)
I20260929 02:00:05.729616 12483 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 02:00:05.733304 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 12483
I20260929 02:00:05.733547 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 02:00:05.734195 12550 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:00:05.743013 12550 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:00:05 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.125@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.125@KRBTEST.COM" created.
I20260929 02:00:05.765573 12550 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } has no permanent_uuid. Determining permanent_uuid...
W20260929 02:00:05.766840 12501 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.125:40905: connect: Connection refused (error 111)
W20260929 02:00:05.768381 12550 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:40905: Network error: Client connection negotiation failed: client connection to 127.0.71.125:40905: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.125@KRBTEST.COM; defaulting to no policy
I20260929 02:00:05.817778 12550 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } attempt: 1
W20260929 02:00:05.819851 12550 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:40905: Network error: Client connection negotiation failed: client connection to 127.0.71.125:40905: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.125@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.125 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.125 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:05.852653 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:40905
--webserver_interface=127.0.71.125
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:36581,127.0.71.85:45153,127.0.71.86:42537,127.0.71.87:34613,127.0.71.88:45991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:42039,127.0.71.125:40905,127.0.71.124:33787 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:00:05.888113 12550 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } attempt: 2
W20260929 02:00:05.890175 12550 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:40905: Network error: Client connection negotiation failed: client connection to 127.0.71.125:40905: connect: Connection refused (error 111)
I20260929 02:00:05.981475 12550 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } attempt: 3
W20260929 02:00:05.983794 12550 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:40905: Network error: Client connection negotiation failed: client connection to 127.0.71.125:40905: connect: Connection refused (error 111)
I20260929 02:00:06.127115 12550 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } attempt: 4
W20260929 02:00:06.129531 12550 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:40905: Network error: Client connection negotiation failed: client connection to 127.0.71.125:40905: connect: Connection refused (error 111)
W20260929 02:00:06.195570 12559 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:06.195891 12559 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:06.196065 12559 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:06.204893 12559 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:00:06.205004 12559 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:06.205101 12559 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:00:06.205195 12559 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:00:06.216353 12559 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:36581,127.0.71.85:45153,127.0.71.86:42537,127.0.71.87:34613,127.0.71.88:45991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:42039,127.0.71.125:40905,127.0.71.124:33787
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.125
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:40905
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:06.217993 12559 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:06.219377 12559 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:06.227767 12564 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:06.227818 12565 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:06.228415 12567 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:06.229451 12559 server_base.cc:1061] running on GCE node
I20260929 02:00:06.230211 12559 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:06.231740 12559 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:06.232970 12559 hybrid_clock.cc:648] HybridClock initialized: now 1790647206232824 us; error 115 us; skew 500 ppm
I20260929 02:00:06.233327 12559 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:06 dist-test-slave-vz23 krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:06.236696 12559 init.cc:377] Logged in from keytab as kudu/127.0.71.125@KRBTEST.COM (short username kudu)
I20260929 02:00:06.238306 12559 webserver.cc:533] Webserver started at http://127.0.71.125:38031/ using document root <none> and password file <none>
I20260929 02:00:06.239030 12559 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:06.239173 12559 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:06.239529 12559 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:06.241971 12559 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-1/data/instance:
uuid: "0d3ea94ffef54f3fab0621aff4dab4df"
format_stamp: "Formatted at 2026-09-29 02:00:06 on dist-test-slave-vz23"
I20260929 02:00:06.242734 12559 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-1/wal/instance:
uuid: "0d3ea94ffef54f3fab0621aff4dab4df"
format_stamp: "Formatted at 2026-09-29 02:00:06 on dist-test-slave-vz23"
I20260929 02:00:06.248103 12559 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.004s
I20260929 02:00:06.251410 12574 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:06.252897 12559 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 02:00:06.253080 12559 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "0d3ea94ffef54f3fab0621aff4dab4df"
format_stamp: "Formatted at 2026-09-29 02:00:06 on dist-test-slave-vz23"
I20260929 02:00:06.253273 12559 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:06.280253 12559 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:06.284108 12559 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:06.284405 12559 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:06.304795 12559 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:40905
I20260929 02:00:06.304900 12625 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:40905 every 8 connection(s)
I20260929 02:00:06.306346 12559 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 02:00:06.310472 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 12559
I20260929 02:00:06.310580 12626 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:00:06.310715 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 02:00:06.316278 12626 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:00:06 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
WARNING: no policy specified for kudu/127.0.71.124@KRBTEST.COM; defaulting to no policy
I20260929 02:00:06.339950 12626 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } has no permanent_uuid. Determining permanent_uuid...
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.124@KRBTEST.COM" created.
Sep 29 02:00:06 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 02:00:06.359285 12626 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } has no permanent_uuid. Determining permanent_uuid...
W20260929 02:00:06.360639 12578 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.124:33787: connect: Connection refused (error 111)
W20260929 02:00:06.362534 12626 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:33787: Network error: Client connection negotiation failed: client connection to 127.0.71.124:33787: connect: Connection refused (error 111)
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.124@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.124@KRBTEST.COM" created.
I20260929 02:00:06.411839 12626 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } attempt: 1
W20260929 02:00:06.414213 12626 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:33787: Network error: Client connection negotiation failed: client connection to 127.0.71.124:33787: connect: Connection refused (error 111)
I20260929 02:00:06.428879 12550 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } attempt: 5
Sep 29 02:00:06 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.124 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.124 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:06.439870 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:33787
--webserver_interface=127.0.71.124
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:36581,127.0.71.85:45153,127.0.71.86:42537,127.0.71.87:34613,127.0.71.88:45991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:42039,127.0.71.125:40905,127.0.71.124:33787 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:00:06.444085 12550 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } has no permanent_uuid. Determining permanent_uuid...
W20260929 02:00:06.446019 12550 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:33787: Network error: Client connection negotiation failed: client connection to 127.0.71.124:33787: connect: Connection refused (error 111)
I20260929 02:00:06.482474 12626 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } attempt: 2
W20260929 02:00:06.484608 12626 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:33787: Network error: Client connection negotiation failed: client connection to 127.0.71.124:33787: connect: Connection refused (error 111)
I20260929 02:00:06.497272 12550 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } attempt: 1
W20260929 02:00:06.499300 12550 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:33787: Network error: Client connection negotiation failed: client connection to 127.0.71.124:33787: connect: Connection refused (error 111)
I20260929 02:00:06.567548 12550 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } attempt: 2
W20260929 02:00:06.569801 12550 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:33787: Network error: Client connection negotiation failed: client connection to 127.0.71.124:33787: connect: Connection refused (error 111)
I20260929 02:00:06.575824 12626 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } attempt: 3
W20260929 02:00:06.577768 12626 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:33787: Network error: Client connection negotiation failed: client connection to 127.0.71.124:33787: connect: Connection refused (error 111)
I20260929 02:00:06.676064 12550 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } attempt: 3
W20260929 02:00:06.678331 12550 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:33787: Network error: Client connection negotiation failed: client connection to 127.0.71.124:33787: connect: Connection refused (error 111)
I20260929 02:00:06.720999 12626 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } attempt: 4
W20260929 02:00:06.723279 12626 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:33787: Network error: Client connection negotiation failed: client connection to 127.0.71.124:33787: connect: Connection refused (error 111)
W20260929 02:00:06.781174 12636 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:06.781535 12636 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:06.781678 12636 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:06.790175 12636 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:00:06.790321 12636 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:06.790436 12636 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:00:06.790525 12636 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:00:06.801534 12636 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:36581,127.0.71.85:45153,127.0.71.86:42537,127.0.71.87:34613,127.0.71.88:45991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-2/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:42039,127.0.71.125:40905,127.0.71.124:33787
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.124
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.124:33787
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
--webserver_interface=127.0.71.124
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:06.803153 12636 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:06.804507 12636 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:06.812633 12642 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:06.812681 12641 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:06.813223 12644 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:06.813979 12636 server_base.cc:1061] running on GCE node
I20260929 02:00:06.814684 12636 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:06.816262 12636 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:06.817492 12636 hybrid_clock.cc:648] HybridClock initialized: now 1790647206817359 us; error 97 us; skew 500 ppm
I20260929 02:00:06.817836 12636 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:06 dist-test-slave-vz23 krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:06.820923 12636 init.cc:377] Logged in from keytab as kudu/127.0.71.124@KRBTEST.COM (short username kudu)
I20260929 02:00:06.822543 12636 webserver.cc:533] Webserver started at http://127.0.71.124:36185/ using document root <none> and password file <none>
I20260929 02:00:06.823338 12636 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:06.823472 12636 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:06.823800 12636 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:06.826223 12636 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-2/data/instance:
uuid: "e9e01a8a110644a8918fce38a4c195d4"
format_stamp: "Formatted at 2026-09-29 02:00:06 on dist-test-slave-vz23"
I20260929 02:00:06.827070 12636 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-2/wal/instance:
uuid: "e9e01a8a110644a8918fce38a4c195d4"
format_stamp: "Formatted at 2026-09-29 02:00:06 on dist-test-slave-vz23"
I20260929 02:00:06.832592 12636 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.005s
I20260929 02:00:06.835834 12651 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:06.837363 12636 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:00:06.837567 12636 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-2/wal
uuid: "e9e01a8a110644a8918fce38a4c195d4"
format_stamp: "Formatted at 2026-09-29 02:00:06 on dist-test-slave-vz23"
I20260929 02:00:06.837754 12636 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:06.851652 12636 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:06.855613 12636 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:06.855568 12550 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } attempt: 4
I20260929 02:00:06.855911 12636 kserver.cc:163] Server-wide thread pool size limit: 3276
W20260929 02:00:06.858012 12550 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.124:33787: Network error: Client connection negotiation failed: client connection to 127.0.71.124:33787: connect: Connection refused (error 111)
I20260929 02:00:06.877189 12636 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.124:33787
I20260929 02:00:06.877215 12702 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.124:33787 every 8 connection(s)
I20260929 02:00:06.878783 12636 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-2/data/info.pb
I20260929 02:00:06.883172 12703 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:00:06.888628 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 12636
I20260929 02:00:06.888816 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-2/wal/instance
I20260929 02:00:06.889151 12703 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:00:06 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 02:00:06 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 02:00:06.912060 12703 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:00:06 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 02:00:06 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 02:00:06.926445 12703 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:00:06 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 02:00:06 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.124@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 02:00:06.944855 12703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4: Bootstrap starting.
I20260929 02:00:06.948021 12703 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4: Neither blocks nor log segments found. Creating new log.
I20260929 02:00:06.949121 12703 log.cc:826] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4: Log is configured to *not* fsync() on all Append() calls
I20260929 02:00:06.952236 12703 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4: No bootstrap required, opened a new log
I20260929 02:00:06.957571 12703 raft_consensus.cc:359] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } }
I20260929 02:00:06.957965 12703 raft_consensus.cc:385] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:00:06.958108 12703 raft_consensus.cc:740] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e9e01a8a110644a8918fce38a4c195d4, State: Initialized, Role: FOLLOWER
I20260929 02:00:06.958724 12703 consensus_queue.cc:260] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } }
I20260929 02:00:06.959581 12713 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } }
I20260929 02:00:06.959838 12713 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:00:06.960613 12703 sys_catalog.cc:565] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:00:06.967940 12636 master_runner.cc:187] Error getting master registration for 127.0.71.126:42039: OK, instance_id { permanent_uuid: "b97759321664486dad92e9d63f3f5332" instance_seqno: 1790647205722186 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 02:00:06.972206 12636 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
W20260929 02:00:06.975153 12724 catalog_manager.cc:1607] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:00:06.975257 12724 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:00:07.022581 12626 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } attempt: 5
Sep 29 02:00:07 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 02:00:07.041102 12626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df: Bootstrap starting.
I20260929 02:00:07.044312 12626 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df: Neither blocks nor log segments found. Creating new log.
I20260929 02:00:07.045380 12626 log.cc:826] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df: Log is configured to *not* fsync() on all Append() calls
I20260929 02:00:07.047802 12626 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df: No bootstrap required, opened a new log
I20260929 02:00:07.053498 12626 raft_consensus.cc:359] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } }
I20260929 02:00:07.053921 12626 raft_consensus.cc:385] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:00:07.054080 12626 raft_consensus.cc:740] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 0d3ea94ffef54f3fab0621aff4dab4df, State: Initialized, Role: FOLLOWER
I20260929 02:00:07.054806 12626 consensus_queue.cc:260] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } }
I20260929 02:00:07.055801 12727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } }
I20260929 02:00:07.056125 12727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:00:07.056797 12626 sys_catalog.cc:565] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [sys.catalog]: configured and running, proceeding with master startup.
W20260929 02:00:07.067494 12738 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:00:07.067673 12738 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:00:07.067862 12727 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:00:07.068136 12727 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } }
I20260929 02:00:07.070099 12727 leader_election.cc:290] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b97759321664486dad92e9d63f3f5332 (127.0.71.126:42039), e9e01a8a110644a8918fce38a4c195d4 (127.0.71.124:33787)
I20260929 02:00:07.072062 12678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9e01a8a110644a8918fce38a4c195d4" is_pre_election: true
I20260929 02:00:07.072655 12678 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 0d3ea94ffef54f3fab0621aff4dab4df in term 0.
I20260929 02:00:07.074436 12578 leader_election.cc:304] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d3ea94ffef54f3fab0621aff4dab4df, e9e01a8a110644a8918fce38a4c195d4; no voters:
I20260929 02:00:07.064707 12559 master_runner.cc:187] Error getting master registration for 127.0.71.126:42039: OK, instance_id { permanent_uuid: "b97759321664486dad92e9d63f3f5332" instance_seqno: 1790647205722186 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 02:00:07.075037 12559 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 02:00:07.075273 12727 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:00:07.075421 12727 raft_consensus.cc:493] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:00:07.075534 12727 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:00:07.079083 12727 raft_consensus.cc:515] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } }
I20260929 02:00:07.079942 12727 leader_election.cc:290] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [CANDIDATE]: Term 1 election: Requested vote from peers b97759321664486dad92e9d63f3f5332 (127.0.71.126:42039), e9e01a8a110644a8918fce38a4c195d4 (127.0.71.124:33787)
I20260929 02:00:07.081055 12678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9e01a8a110644a8918fce38a4c195d4"
I20260929 02:00:07.081265 12678 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:00:07.084898 12678 raft_consensus.cc:2468] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d3ea94ffef54f3fab0621aff4dab4df in term 1.
I20260929 02:00:07.085536 12578 leader_election.cc:304] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 0d3ea94ffef54f3fab0621aff4dab4df, e9e01a8a110644a8918fce38a4c195d4; no voters:
I20260929 02:00:07.086017 12727 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:00:07.086884 12727 raft_consensus.cc:697] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [term 1 LEADER]: Becoming Leader. State: Replica: 0d3ea94ffef54f3fab0621aff4dab4df, State: Running, Role: LEADER
I20260929 02:00:07.087915 12727 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } }
I20260929 02:00:07.093770 12740 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d3ea94ffef54f3fab0621aff4dab4df. Latest consensus state: current_term: 1 leader_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } }
I20260929 02:00:07.094115 12740 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:07.106750 12742 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
W20260929 02:00:07.108278 12657 tablet.cc:2433] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:00:07.109452 12742 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 02:00:07.124527 12678 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4 [term 1 FOLLOWER]: Refusing update from remote peer 0d3ea94ffef54f3fab0621aff4dab4df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:00:07.125582 12740 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:00:07.135221 12550 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } attempt: 5
I20260929 02:00:07.142284 12713 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d3ea94ffef54f3fab0621aff4dab4df. Latest consensus state: current_term: 1 leader_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } }
I20260929 02:00:07.142628 12713 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:00:07.144529 12743 mvcc.cc:204] Tried to move back new op lower bound from 7334490960367149056 to 7334490960242245632. Current Snapshot: MvccSnapshot[applied={T|T < 7334490960367149056}]
Sep 29 02:00:07 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647205, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 02:00:07.152316 12744 mvcc.cc:204] Tried to move back new op lower bound from 7334490960367149056 to 7334490960242245632. Current Snapshot: MvccSnapshot[applied={T|T < 7334490960367149056}]
I20260929 02:00:07.157801 12713 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } }
I20260929 02:00:07.158028 12713 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:00:07.161871 12550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332: Bootstrap starting.
I20260929 02:00:07.166467 12550 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332: Neither blocks nor log segments found. Creating new log.
I20260929 02:00:07.167919 12550 log.cc:826] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332: Log is configured to *not* fsync() on all Append() calls
I20260929 02:00:07.169068 12727 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } }
I20260929 02:00:07.169445 12727 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:07.171340 12742 catalog_manager.cc:1396] Generated new cluster ID: 3145234aa8a04216b43cf8dd7e4989b8
I20260929 02:00:07.171450 12742 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 02:00:07.177323 12550 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332: No bootstrap required, opened a new log
I20260929 02:00:07.193276 12742 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 02:00:07.195192 12742 catalog_manager.cc:1553] Loading token signing keys...
I20260929 02:00:07.202682 12550 raft_consensus.cc:359] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } }
I20260929 02:00:07.203684 12550 raft_consensus.cc:385] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:00:07.204108 12550 raft_consensus.cc:740] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: b97759321664486dad92e9d63f3f5332, State: Initialized, Role: FOLLOWER
I20260929 02:00:07.206084 12550 consensus_queue.cc:260] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } }
I20260929 02:00:07.228763 12742 catalog_manager.cc:6219] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df: Generated new TSK 0
I20260929 02:00:07.229928 12742 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 02:00:07.229998 12524 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b97759321664486dad92e9d63f3f5332"
I20260929 02:00:07.230818 12525 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b97759321664486dad92e9d63f3f5332" is_pre_election: true
I20260929 02:00:07.232199 12524 raft_consensus.cc:3060] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:00:07.232559 12550 sys_catalog.cc:565] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:00:07.236824 12524 raft_consensus.cc:2468] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 0d3ea94ffef54f3fab0621aff4dab4df in term 1.
I20260929 02:00:07.238564 12523 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [term 1 FOLLOWER]: Refusing update from remote peer 0d3ea94ffef54f3fab0621aff4dab4df: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:00:07.239929 12740 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [LEADER]: Connected to new peer: Peer: permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:00:07.251619 12750 sys_catalog.cc:455] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } }
I20260929 02:00:07.252413 12750 sys_catalog.cc:458] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:00:07.273434 12753 mvcc.cc:204] Tried to move back new op lower bound from 7334490960367149056 to 7334490960242245632. Current Snapshot: MvccSnapshot[applied={T|T < 7334490960367149056}]
I20260929 02:00:07.280752 12752 sys_catalog.cc:455] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } }
I20260929 02:00:07.281184 12752 sys_catalog.cc:458] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:00:07.281647 12750 sys_catalog.cc:455] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d3ea94ffef54f3fab0621aff4dab4df. Latest consensus state: current_term: 1 leader_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } }
I20260929 02:00:07.281903 12750 sys_catalog.cc:458] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [sys.catalog]: This master's current role is: FOLLOWER
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
W20260929 02:00:07.367543 285 mini_kdc.cc:268] Time spent creating service keytab for kudu/127.0.71.65: real 0.117s user 0.000s sys 0.026s
I20260929 02:00:07.382206 12768 catalog_manager.cc:1308] Loaded cluster ID: 3145234aa8a04216b43cf8dd7e4989b8
I20260929 02:00:07.382328 12768 catalog_manager.cc:1601] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332: loading cluster ID for follower catalog manager: success
I20260929 02:00:07.390374 12768 catalog_manager.cc:1623] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332: acquiring CA information for follower catalog manager: success
I20260929 02:00:07.393205 12768 catalog_manager.cc:1651] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:07.452809 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:42039,127.0.71.125:40905,127.0.71.124:33787
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:36581,127.0.71.85:45153,127.0.71.86:42537,127.0.71.87:34613,127.0.71.88:45991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:00:07.869628 12773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } }
I20260929 02:00:07.870049 12773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [sys.catalog]: This master's current role is: LEADER
W20260929 02:00:07.914330 12772 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:07.914615 12772 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:07.914825 12772 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:07.923297 12772 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:07.923528 12772 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 02:00:07.934599 12772 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:36581,127.0.71.85:45153,127.0.71.86:42537,127.0.71.87:34613,127.0.71.88:45991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:42039,127.0.71.125:40905,127.0.71.124:33787
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:07.936107 12772 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:07.937422 12772 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:07.946718 12781 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:07.947194 12782 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:07.948414 12772 server_base.cc:1061] running on GCE node
W20260929 02:00:07.948585 12784 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:07.949196 12772 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:07.950619 12772 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:07.951838 12772 hybrid_clock.cc:648] HybridClock initialized: now 1790647207951708 us; error 93 us; skew 500 ppm
I20260929 02:00:07.952157 12772 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:07 dist-test-slave-vz23 krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:07.955892 12772 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 02:00:07.957793 12772 webserver.cc:533] Webserver started at http://127.0.71.65:33807/ using document root <none> and password file <none>
I20260929 02:00:07.958549 12772 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:07.958645 12772 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:07.958938 12772 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:07.961568 12772 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "6ab6889a9df84d01bf9a0ebd21787d5f"
format_stamp: "Formatted at 2026-09-29 02:00:07 on dist-test-slave-vz23"
I20260929 02:00:07.962343 12772 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "6ab6889a9df84d01bf9a0ebd21787d5f"
format_stamp: "Formatted at 2026-09-29 02:00:07 on dist-test-slave-vz23"
I20260929 02:00:07.967810 12772 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 02:00:07.971257 12791 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:07.972782 12772 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:00:07.972994 12772 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "6ab6889a9df84d01bf9a0ebd21787d5f"
format_stamp: "Formatted at 2026-09-29 02:00:07 on dist-test-slave-vz23"
I20260929 02:00:07.973205 12772 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:07.977121 12724 catalog_manager.cc:1308] Loaded cluster ID: 3145234aa8a04216b43cf8dd7e4989b8
I20260929 02:00:07.977206 12724 catalog_manager.cc:1601] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4: loading cluster ID for follower catalog manager: success
I20260929 02:00:07.980125 12724 catalog_manager.cc:1623] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4: acquiring CA information for follower catalog manager: success
I20260929 02:00:07.981901 12724 catalog_manager.cc:1651] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 02:00:07.996352 12772 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:08.000396 12772 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:08.000771 12772 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:08.001741 12772 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 02:00:08.003957 12772 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:00:08.004048 12772 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:08.004185 12772 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 02:00:08.004247 12772 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:08.050266 12772 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:41917
I20260929 02:00:08.050385 12903 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:41917 every 8 connection(s)
I20260929 02:00:08.051707 12772 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 02:00:08.053642 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 12772
I20260929 02:00:08.053859 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
Sep 29 02:00:08 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 02:00:08 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 02:00:08 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647207, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 02:00:08.090840 12904 heartbeater.cc:344] Connected to a master server at 127.0.71.124:33787
I20260929 02:00:08.091252 12904 heartbeater.cc:461] Registering TS with master...
I20260929 02:00:08.092286 12904 heartbeater.cc:507] Master 127.0.71.124:33787 requested a full tablet report, sending...
I20260929 02:00:08.093623 12906 heartbeater.cc:344] Connected to a master server at 127.0.71.125:40905
I20260929 02:00:08.093770 12906 heartbeater.cc:461] Registering TS with master...
I20260929 02:00:08.094251 12906 heartbeater.cc:507] Master 127.0.71.125:40905 requested a full tablet report, sending...
I20260929 02:00:08.094897 12668 ts_manager.cc:194] Registered new tserver with Master: 6ab6889a9df84d01bf9a0ebd21787d5f (127.0.71.65:41917)
I20260929 02:00:08.095681 12591 ts_manager.cc:194] Registered new tserver with Master: 6ab6889a9df84d01bf9a0ebd21787d5f (127.0.71.65:41917)
I20260929 02:00:08.095825 12905 heartbeater.cc:344] Connected to a master server at 127.0.71.126:42039
I20260929 02:00:08.095971 12905 heartbeater.cc:461] Registering TS with master...
I20260929 02:00:08.096417 12905 heartbeater.cc:507] Master 127.0.71.126:42039 requested a full tablet report, sending...
I20260929 02:00:08.097662 12591 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:36937
I20260929 02:00:08.098179 12515 ts_manager.cc:194] Registered new tserver with Master: 6ab6889a9df84d01bf9a0ebd21787d5f (127.0.71.65:41917)
I20260929 02:00:08.112684 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
WARNING: no policy specified for kudu/127.0.71.123@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.123@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.123 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.123 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.123@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.123@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.123 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.123 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:08.224738 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-3/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-3/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-3/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.123:40849
--webserver_interface=127.0.71.123
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.123
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:36581,127.0.71.85:45153,127.0.71.86:42537,127.0.71.87:34613,127.0.71.88:45991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:42039,127.0.71.125:40905,127.0.71.124:33787,127.0.71.123:40849 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:08.562079 12918 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:08.562407 12918 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:08.562557 12918 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:08.570986 12918 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:00:08.571084 12918 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:08.571214 12918 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:00:08.571295 12918 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:00:08.582180 12918 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-3/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:36581,127.0.71.85:45153,127.0.71.86:42537,127.0.71.87:34613,127.0.71.88:45991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-3/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-3/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:42039,127.0.71.125:40905,127.0.71.124:33787,127.0.71.123:40849
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.123
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.123:40849
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
--webserver_interface=127.0.71.123
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:08.583738 12918 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:08.585069 12918 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:08.592878 12923 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:08.593098 12924 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:08.593885 12926 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:08.594079 12918 server_base.cc:1061] running on GCE node
I20260929 02:00:08.594831 12918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:08.596272 12918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:08.597505 12918 hybrid_clock.cc:648] HybridClock initialized: now 1790647208597376 us; error 99 us; skew 500 ppm
I20260929 02:00:08.597856 12918 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:08 dist-test-slave-vz23 krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647208, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:08.608315 12918 init.cc:377] Logged in from keytab as kudu/127.0.71.123@KRBTEST.COM (short username kudu)
I20260929 02:00:08.610169 12918 webserver.cc:533] Webserver started at http://127.0.71.123:35969/ using document root <none> and password file <none>
I20260929 02:00:08.610888 12918 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:08.611021 12918 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:08.611392 12918 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:08.614012 12918 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-3/data/instance:
uuid: "18262cb753d041a788c761bb04bb2a52"
format_stamp: "Formatted at 2026-09-29 02:00:08 on dist-test-slave-vz23"
I20260929 02:00:08.614801 12918 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-3/wal/instance:
uuid: "18262cb753d041a788c761bb04bb2a52"
format_stamp: "Formatted at 2026-09-29 02:00:08 on dist-test-slave-vz23"
I20260929 02:00:08.620257 12918 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 02:00:08.624300 12933 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:08.625721 12918 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:00:08.625905 12918 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "18262cb753d041a788c761bb04bb2a52"
format_stamp: "Formatted at 2026-09-29 02:00:08 on dist-test-slave-vz23"
I20260929 02:00:08.626113 12918 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:08.647492 12918 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:08.651491 12918 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:08.651788 12918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:08.673966 12918 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.123:40849
I20260929 02:00:08.674005 12984 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.123:40849 every 8 connection(s)
I20260929 02:00:08.675478 12918 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
I20260929 02:00:08.679854 12985 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:00:08.682024 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 12918
I20260929 02:00:08.686980 12985 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:00:08 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
Sep 29 02:00:08 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647208, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 02:00:08.705192 12985 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:00:08 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647208, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
I20260929 02:00:08.718235 12985 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:00:08 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647208, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
I20260929 02:00:08.732081 12985 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.123" port: 40849 } has no permanent_uuid. Determining permanent_uuid...
Sep 29 02:00:08 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647208, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
I20260929 02:00:08.758814 12985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52: Bootstrap starting.
I20260929 02:00:08.762696 12985 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52: Neither blocks nor log segments found. Creating new log.
I20260929 02:00:08.764171 12985 log.cc:826] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52: Log is configured to *not* fsync() on all Append() calls
I20260929 02:00:08.767239 12985 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52: No bootstrap required, opened a new log
I20260929 02:00:08.773736 12985 raft_consensus.cc:359] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 40849 } }
I20260929 02:00:08.774060 12985 raft_consensus.cc:385] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:00:08.774161 12985 raft_consensus.cc:740] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 18262cb753d041a788c761bb04bb2a52, State: Initialized, Role: FOLLOWER
I20260929 02:00:08.775041 12985 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 40849 } }
I20260929 02:00:08.776113 12994 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 40849 } } }
I20260929 02:00:08.776696 12994 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:00:08.777263 12985 sys_catalog.cc:565] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 02:00:08.788051 13005 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:00:08.788190 13005 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:00:08.789129 12994 raft_consensus.cc:493] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:00:08.789390 12994 raft_consensus.cc:515] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 40849 } }
I20260929 02:00:08.791327 12994 leader_election.cc:290] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers b97759321664486dad92e9d63f3f5332 (127.0.71.126:42039), 0d3ea94ffef54f3fab0621aff4dab4df (127.0.71.125:40905), e9e01a8a110644a8918fce38a4c195d4 (127.0.71.124:33787)
I20260929 02:00:08.791901 12601 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "18262cb753d041a788c761bb04bb2a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" is_pre_election: true
I20260929 02:00:08.793273 12678 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "18262cb753d041a788c761bb04bb2a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "e9e01a8a110644a8918fce38a4c195d4" is_pre_election: true
I20260929 02:00:08.793349 12523 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "18262cb753d041a788c761bb04bb2a52" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "b97759321664486dad92e9d63f3f5332" is_pre_election: true
I20260929 02:00:08.794214 12934 leader_election.cc:304] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 4 voters: 1 yes votes; 2 no votes. yes voters: 18262cb753d041a788c761bb04bb2a52; no voters: 0d3ea94ffef54f3fab0621aff4dab4df, b97759321664486dad92e9d63f3f5332
I20260929 02:00:08.794821 12994 raft_consensus.cc:2749] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:00:08.798512 12918 master_runner.cc:429] Detected that this master 18262cb753d041a788c761bb04bb2a52 is joining an existing cluster
I20260929 02:00:08.798645 12918 catalog_manager.cc:1037] Shutting down the Catalog Manager
I20260929 02:00:08.799481 12918 raft_consensus.cc:2243] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52 [term 0 FOLLOWER]: Raft consensus shutting down.
I20260929 02:00:08.799813 12918 raft_consensus.cc:2272] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52 [term 0 FOLLOWER]: Raft consensus is shut down!
I20260929 02:00:08.800017 12918 tablet_replica.cc:333] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52: stopping tablet replica
I20260929 02:00:08.801508 12918 master_runner.cc:434] Initiating AddMaster RPC to add 127.0.71.123:40849
Sep 29 02:00:08 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647206, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.125@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
I20260929 02:00:08.857757 12591 catalog_manager.cc:7351] Initiating ChangeConfig request to add master 127.0.71.123:40849: tablet_id: "00000000000000000000000000000000"
type: ADD_PEER
server {
permanent_uuid: "18262cb753d041a788c761bb04bb2a52"
member_type: NON_VOTER
last_known_addr {
host: "127.0.71.123"
port: 40849
}
attrs {
promote: true
}
}
dest_uuid: "0d3ea94ffef54f3fab0621aff4dab4df"
cas_config_opid_index: -1
I20260929 02:00:08.859198 12591 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 4, Committed index: 4, Last appended: 1.4, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: true } }
I20260929 02:00:08.862439 12523 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [term 1 FOLLOWER]: Refusing update from remote peer 0d3ea94ffef54f3fab0621aff4dab4df: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
I20260929 02:00:08.862629 12678 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4 [term 1 FOLLOWER]: Refusing update from remote peer 0d3ea94ffef54f3fab0621aff4dab4df: Log matching property violated. Preceding OpId in replica: term: 1 index: 4. Preceding OpId from leader: term: 1 index: 5. (index mismatch)
W20260929 02:00:08.863124 12578 consensus_peers.cc:597] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df -> Peer 18262cb753d041a788c761bb04bb2a52 (127.0.71.123:40849): Couldn't send request to peer 18262cb753d041a788c761bb04bb2a52. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: SHUTDOWN. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:00:08.863314 13018 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [LEADER]: Connected to new peer: Peer: permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 02:00:08.863579 12775 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 5, Last known committed idx: 4, Time since last communication: 0.000s
I20260929 02:00:08.866683 12775 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [term 1 LEADER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 18262cb753d041a788c761bb04bb2a52 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: true } } }
I20260929 02:00:08.867370 12678 raft_consensus.cc:2955] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 18262cb753d041a788c761bb04bb2a52 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: true } } }
I20260929 02:00:08.871110 12775 catalog_manager.cc:7355] Successfully completed master ChangeConfig request to add master 127.0.71.123:40849
I20260929 02:00:08.872030 12918 master.cc:562] Master@127.0.71.123:40849 shutting down...
I20260929 02:00:08.885815 12523 raft_consensus.cc:2955] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [term 1 FOLLOWER]: Committing config change with OpId 1.5: config changed from index -1 to 5, NON_VOTER 18262cb753d041a788c761bb04bb2a52 (127.0.71.123) added. New config: { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: true } } }
I20260929 02:00:08.898651 13024 sys_catalog.cc:455] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: true } } }
I20260929 02:00:08.898989 13024 sys_catalog.cc:458] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:00:08.900435 12918 master.cc:584] Master@127.0.71.123:40849 shutdown complete.
I20260929 02:00:08.900647 12918 master_runner.cc:306] Clearing existing system tablet
I20260929 02:00:08.901206 12918 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:08.901319 12918 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:08.908854 12918 fs_manager.cc:714] Time spent opening directory manager: real 0.006s user 0.000s sys 0.002s
I20260929 02:00:08.910825 13022 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: true } } }
I20260929 02:00:08.911533 13022 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:00:08.912772 13026 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:08.913228 12918 fs_report.cc:389] FS layout report
--------------------
wal directory:
metadata directory:
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:08.913549 12918 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.001s sys 0.001s
I20260929 02:00:08.913636 12918 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "18262cb753d041a788c761bb04bb2a52"
format_stamp: "Formatted at 2026-09-29 02:00:08 on dist-test-slave-vz23"
I20260929 02:00:08.914405 13021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: true } } }
I20260929 02:00:08.914424 13018 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: true } } }
I20260929 02:00:08.914738 13021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:08.914775 13018 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:08.915387 13021 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: true } } }
I20260929 02:00:08.915555 13021 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:08.918290 12918 ts_tablet_manager.cc:1962] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:00:08.925299 12918 ts_tablet_manager.cc:1975] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId unknown
I20260929 02:00:08.925470 12918 log.cc:1199] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-3/wal/wals/00000000000000000000000000000000
I20260929 02:00:08.925899 12918 ts_tablet_manager.cc:1996] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52: Deleting consensus metadata
I20260929 02:00:08.926152 12918 master_runner.cc:316] Copying system tablet from 127.0.71.125:40905
I20260929 02:00:08.928910 12918 tablet_copy_client.cc:323] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52: tablet copy: Beginning tablet copy session from remote peer at address 127.0.71.125:40905
I20260929 02:00:08.944278 12611 tablet_copy_service.cc:140] P 0d3ea94ffef54f3fab0621aff4dab4df: Received BeginTabletCopySession request for tablet 00000000000000000000000000000000 from peer 18262cb753d041a788c761bb04bb2a52 ({username='kudu', principal='kudu/127.0.71.123@KRBTEST.COM'} at 127.0.0.1:41958)
I20260929 02:00:08.944563 12611 tablet_copy_service.cc:161] P 0d3ea94ffef54f3fab0621aff4dab4df: Beginning new tablet copy session on tablet 00000000000000000000000000000000 from peer 18262cb753d041a788c761bb04bb2a52 at {username='kudu', principal='kudu/127.0.71.123@KRBTEST.COM'} at 127.0.0.1:41958: session id = 18262cb753d041a788c761bb04bb2a52-00000000000000000000000000000000
I20260929 02:00:08.947333 12611 tablet_copy_source_session.cc:215] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df: Tablet Copy: opened 0 blocks and 1 log segments
I20260929 02:00:08.949329 12918 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total
I20260929 02:00:08.953995 12918 tablet_copy_client.cc:806] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52: tablet copy: Starting download of 0 data blocks...
I20260929 02:00:08.954280 12918 tablet_copy_client.cc:670] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52: tablet copy: Starting download of 1 WAL segments...
I20260929 02:00:08.956921 12918 tablet_copy_client.cc:538] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52: tablet copy: Tablet Copy complete. Replacing tablet superblock.
I20260929 02:00:08.960489 12611 tablet_copy_service.cc:342] P 0d3ea94ffef54f3fab0621aff4dab4df: Request end of tablet copy session 18262cb753d041a788c761bb04bb2a52-00000000000000000000000000000000 received from {username='kudu', principal='kudu/127.0.71.123@KRBTEST.COM'} at 127.0.0.1:41958
I20260929 02:00:08.960688 12611 tablet_copy_service.cc:434] P 0d3ea94ffef54f3fab0621aff4dab4df: ending tablet copy session 18262cb753d041a788c761bb04bb2a52-00000000000000000000000000000000 on tablet 00000000000000000000000000000000 with peer 18262cb753d041a788c761bb04bb2a52
W20260929 02:00:08.967193 12918 builtin_ntp.cc:688] coult not shutdown socket: Network error: shutdown error: Transport endpoint is not connected (error 107)
I20260929 02:00:08.970418 12918 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:08.973055 13034 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:08.973548 13035 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:08.975143 13037 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:08.975657 12918 server_base.cc:1061] running on GCE node
I20260929 02:00:08.975970 12918 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:08.976722 12918 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:08.977890 12918 hybrid_clock.cc:648] HybridClock initialized: now 1790647208977759 us; error 101 us; skew 500 ppm
I20260929 02:00:08.978139 12918 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:08 dist-test-slave-vz23 krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647208, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.123@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:08.980396 12918 init.cc:377] Logged in from keytab as kudu/127.0.71.123@KRBTEST.COM (short username kudu)
I20260929 02:00:08.981568 12918 webserver.cc:533] Webserver started at http://127.0.71.123:40799/ using document root <none> and password file <none>
I20260929 02:00:08.981913 12918 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:08.982018 12918 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:08.984985 12918 fs_manager.cc:714] Time spent opening directory manager: real 0.002s user 0.002s sys 0.001s
I20260929 02:00:08.987030 13044 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:08.987452 12918 fs_manager.cc:730] Time spent opening block manager: real 0.001s user 0.001s sys 0.001s
I20260929 02:00:08.987569 12918 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-3/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-3/wal
uuid: "18262cb753d041a788c761bb04bb2a52"
format_stamp: "Formatted at 2026-09-29 02:00:08 on dist-test-slave-vz23"
I20260929 02:00:08.987735 12918 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-3/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-3/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-3/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:09.012029 12918 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:09.016863 12918 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:09.039284 12918 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.123:40849
I20260929 02:00:09.039328 13095 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.123:40849 every 8 connection(s)
I20260929 02:00:09.040495 12918 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-3/data/info.pb
I20260929 02:00:09.044049 13096 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:00:09.046391 13096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52: Bootstrap starting.
I20260929 02:00:09.060716 13096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:00:09.061157 13096 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52: Bootstrap complete.
I20260929 02:00:09.062145 13096 raft_consensus.cc:359] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52 [term 1 LEARNER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: true } }
I20260929 02:00:09.062322 13096 raft_consensus.cc:740] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52 [term 1 LEARNER]: Becoming Follower/Learner. State: Replica: 18262cb753d041a788c761bb04bb2a52, State: Initialized, Role: LEARNER
I20260929 02:00:09.062559 13096 consensus_queue.cc:260] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: true } }
I20260929 02:00:09.063603 13096 sys_catalog.cc:565] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:00:09.063382 13099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: true } } }
I20260929 02:00:09.063735 13099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52 [sys.catalog]: This master's current role is: LEARNER
I20260929 02:00:09.077916 13111 catalog_manager.cc:1308] Loaded cluster ID: 3145234aa8a04216b43cf8dd7e4989b8
I20260929 02:00:09.078081 13111 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52: loading cluster ID for follower catalog manager: success
I20260929 02:00:09.081897 13111 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52: acquiring CA information for follower catalog manager: success
I20260929 02:00:09.083922 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 12772
I20260929 02:00:09.084685 13111 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
Sep 29 02:00:09 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 02:00:09.100101 12906 heartbeater.cc:499] Master 127.0.71.125:40905 was elected leader, sending a full tablet report...
I20260929 02:00:09.101749 12899 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:00:09.191291 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 12772
I20260929 02:00:09.214852 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:41917
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:42039,127.0.71.125:40905,127.0.71.124:33787
--webserver_port=33807
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:36581,127.0.71.85:45153,127.0.71.86:42537,127.0.71.87:34613,127.0.71.88:45991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:42039,127.0.71.125:40905,127.0.71.124:33787,127.0.71.123:40849 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:00:09.271020 13071 raft_consensus.cc:1217] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52 [term 1 LEARNER]: Deduplicated request from leader. Original: 1.4->[1.5-1.5] Dedup: 1.5->[]
I20260929 02:00:09.271852 13099 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d3ea94ffef54f3fab0621aff4dab4df. Latest consensus state: current_term: 1 leader_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: true } } }
I20260929 02:00:09.272147 13099 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52 [sys.catalog]: This master's current role is: LEARNER
W20260929 02:00:09.559370 13114 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:09.559674 13114 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:09.559792 13114 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:09.568822 13114 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:09.569018 13114 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 02:00:09.580019 13114 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:36581,127.0.71.85:45153,127.0.71.86:42537,127.0.71.87:34613,127.0.71.88:45991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:41917
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=33807
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:42039,127.0.71.125:40905,127.0.71.124:33787,127.0.71.123:40849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:09.581543 13114 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:09.582824 13114 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:09.592118 13120 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:09.592219 13119 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:09.592931 13122 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:09.594367 13114 server_base.cc:1061] running on GCE node
I20260929 02:00:09.594933 13114 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:09.596061 13114 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:09.597285 13114 hybrid_clock.cc:648] HybridClock initialized: now 1790647209597133 us; error 109 us; skew 500 ppm
I20260929 02:00:09.597631 13114 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:09 dist-test-slave-vz23 krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:09.601018 13114 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 02:00:09.602771 13114 webserver.cc:533] Webserver started at http://127.0.71.65:33807/ using document root <none> and password file <none>
I20260929 02:00:09.603432 13114 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:09.603543 13114 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:09.608984 13114 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 02:00:09.611855 13129 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:09.613453 13114 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:00:09.613624 13114 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "6ab6889a9df84d01bf9a0ebd21787d5f"
format_stamp: "Formatted at 2026-09-29 02:00:07 on dist-test-slave-vz23"
I20260929 02:00:09.614428 13114 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:09.635151 13114 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:09.639245 13114 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:09.639545 13114 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:09.640666 13114 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 02:00:09.642228 13114 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:00:09.642345 13114 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:09.642473 13114 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 02:00:09.642557 13114 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:09.678529 13114 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:41917
I20260929 02:00:09.678619 13241 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:41917 every 8 connection(s)
I20260929 02:00:09.680137 13114 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 02:00:09.681879 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 13114
Sep 29 02:00:09 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
Sep 29 02:00:09 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 02:00:09 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 02:00:09 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647209, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
I20260929 02:00:09.725268 13243 heartbeater.cc:344] Connected to a master server at 127.0.71.126:42039
I20260929 02:00:09.725669 13243 heartbeater.cc:461] Registering TS with master...
I20260929 02:00:09.726828 13243 heartbeater.cc:507] Master 127.0.71.126:42039 requested a full tablet report, sending...
I20260929 02:00:09.728871 13242 heartbeater.cc:344] Connected to a master server at 127.0.71.124:33787
I20260929 02:00:09.728904 12515 ts_manager.cc:194] Re-registered known tserver with Master: 6ab6889a9df84d01bf9a0ebd21787d5f (127.0.71.65:41917)
I20260929 02:00:09.728996 13242 heartbeater.cc:461] Registering TS with master...
I20260929 02:00:09.729487 13242 heartbeater.cc:507] Master 127.0.71.124:33787 requested a full tablet report, sending...
I20260929 02:00:09.730772 12667 ts_manager.cc:194] Re-registered known tserver with Master: 6ab6889a9df84d01bf9a0ebd21787d5f (127.0.71.65:41917)
I20260929 02:00:09.731930 13244 heartbeater.cc:344] Connected to a master server at 127.0.71.123:40849
I20260929 02:00:09.732092 13244 heartbeater.cc:461] Registering TS with master...
I20260929 02:00:09.732235 13245 heartbeater.cc:344] Connected to a master server at 127.0.71.125:40905
I20260929 02:00:09.732501 13245 heartbeater.cc:461] Registering TS with master...
I20260929 02:00:09.732752 13244 heartbeater.cc:507] Master 127.0.71.123:40849 requested a full tablet report, sending...
I20260929 02:00:09.732983 13245 heartbeater.cc:507] Master 127.0.71.125:40905 requested a full tablet report, sending...
I20260929 02:00:09.734131 12591 ts_manager.cc:194] Re-registered known tserver with Master: 6ab6889a9df84d01bf9a0ebd21787d5f (127.0.71.65:41917)
I20260929 02:00:09.734522 13061 ts_manager.cc:194] Registered new tserver with Master: 6ab6889a9df84d01bf9a0ebd21787d5f (127.0.71.65:41917)
I20260929 02:00:09.735270 12591 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:53945
I20260929 02:00:09.737824 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 02:00:09.746337 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 02:00:09.746510 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 12483
I20260929 02:00:09.747377 12545 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:00:09.813084 12775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: true } } }
I20260929 02:00:09.813380 12775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:09.815361 13256 raft_consensus.cc:1064] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df: attempting to promote NON_VOTER 18262cb753d041a788c761bb04bb2a52 to VOTER
I20260929 02:00:09.816387 13256 consensus_queue.cc:237] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 5, Committed index: 5, Last appended: 1.5, Last appended by leader: 0, Current term: 1, Majority size: 3, State: 0, Mode: LEADER, active raft config: opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: false } }
I20260929 02:00:09.819283 13071 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52 [term 1 LEARNER]: Refusing update from remote peer 0d3ea94ffef54f3fab0621aff4dab4df: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 02:00:09.819866 12678 raft_consensus.cc:1275] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4 [term 1 FOLLOWER]: Refusing update from remote peer 0d3ea94ffef54f3fab0621aff4dab4df: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 02:00:09.820009 13256 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [LEADER]: Connected to new peer: Peer: permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: false }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 02:00:09.820576 13258 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [LEADER]: Connected to new peer: Peer: permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
I20260929 02:00:09.823890 13256 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [term 1 LEADER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 18262cb753d041a788c761bb04bb2a52 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: false } } }
I20260929 02:00:09.824687 13071 raft_consensus.cc:2955] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 18262cb753d041a788c761bb04bb2a52 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: false } } }
I20260929 02:00:09.827354 12775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [sys.catalog]: SysCatalogTable state changed. Reason: Config change replication complete. Latest consensus state: current_term: 1 leader_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: false } } }
I20260929 02:00:09.827626 12775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:09.827894 12775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: false } } }
I20260929 02:00:09.828104 12775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:09.827929 12678 raft_consensus.cc:2955] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 18262cb753d041a788c761bb04bb2a52 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: false } } }
I20260929 02:00:09.828332 12775 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: false } } }
I20260929 02:00:09.828538 12775 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:09.831372 13261 sys_catalog.cc:455] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: false } } }
I20260929 02:00:09.831749 13261 sys_catalog.cc:458] T 00000000000000000000000000000000 P e9e01a8a110644a8918fce38a4c195d4 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:00:09.832242 13260 sys_catalog.cc:455] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: false } } }
I20260929 02:00:09.832624 13260 sys_catalog.cc:458] T 00000000000000000000000000000000 P 18262cb753d041a788c761bb04bb2a52 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:00:09.866910 12523 raft_consensus.cc:1275] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [term 1 FOLLOWER]: Refusing update from remote peer 0d3ea94ffef54f3fab0621aff4dab4df: Log matching property violated. Preceding OpId in replica: term: 1 index: 5. Preceding OpId from leader: term: 1 index: 6. (index mismatch)
I20260929 02:00:09.867153 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 12483
I20260929 02:00:09.868130 13256 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [LEADER]: Connected to new peer: Peer: permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 6, Last known committed idx: 5, Time since last communication: 0.000s
W20260929 02:00:09.890898 12575 connection.cc:573] client connection to 127.0.71.126:42039 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 02:00:09.891753 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:42039
--webserver_interface=127.0.71.126
--webserver_port=42021
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:36581,127.0.71.85:45153,127.0.71.86:42537,127.0.71.87:34613,127.0.71.88:45991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:42039,127.0.71.125:40905,127.0.71.124:33787
--master_addresses=127.0.71.126:42039,127.0.71.125:40905,127.0.71.124:33787,127.0.71.123:40849 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:09.897225 12575 consensus_peers.cc:597] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df -> Peer b97759321664486dad92e9d63f3f5332 (127.0.71.126:42039): Couldn't send request to peer b97759321664486dad92e9d63f3f5332. Status: Network error: Client connection negotiation failed: client connection to 127.0.71.126:42039: connect: Connection refused (error 111). This is attempt 1: this message will repeat every 5th retry.
W20260929 02:00:10.225404 13264 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:10.225749 13264 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:10.225920 13264 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:10.234709 13264 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:00:10.234817 13264 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:10.234905 13264 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:00:10.234988 13264 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:00:10.245980 13264 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:36581,127.0.71.85:45153,127.0.71.86:42537,127.0.71.87:34613,127.0.71.88:45991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:42039,127.0.71.125:40905,127.0.71.124:33787,127.0.71.123:40849
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:42039
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=42021
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:10.247577 13264 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:10.248943 13264 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:10.257715 13269 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:10.257797 13272 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:10.257776 13270 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:10.258976 13264 server_base.cc:1061] running on GCE node
I20260929 02:00:10.259763 13264 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:10.261479 13264 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:10.262704 13264 hybrid_clock.cc:648] HybridClock initialized: now 1790647210262568 us; error 107 us; skew 500 ppm
I20260929 02:00:10.263047 13264 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:10 dist-test-slave-vz23 krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:10.266333 13264 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 02:00:10.267977 13264 webserver.cc:533] Webserver started at http://127.0.71.126:42021/ using document root <none> and password file <none>
I20260929 02:00:10.268764 13264 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:10.268905 13264 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:10.274391 13264 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.005s sys 0.000s
I20260929 02:00:10.277168 13279 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:10.278712 13264 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:00:10.278905 13264 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "b97759321664486dad92e9d63f3f5332"
format_stamp: "Formatted at 2026-09-29 02:00:05 on dist-test-slave-vz23"
I20260929 02:00:10.279680 13264 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:10.318894 13264 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:10.322986 13264 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:10.323261 13264 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:10.344751 13264 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:42039
I20260929 02:00:10.344770 13330 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:42039 every 8 connection(s)
I20260929 02:00:10.346408 13264 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 02:00:10.347793 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 13264
I20260929 02:00:10.348165 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 13114
I20260929 02:00:10.353900 13331 sys_catalog.cc:263] Verifying existing consensus state
I20260929 02:00:10.357545 13331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332: Bootstrap starting.
I20260929 02:00:10.363994 13237 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:00:10.380558 13331 log.cc:826] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332: Log is configured to *not* fsync() on all Append() calls
I20260929 02:00:10.391772 13331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332: Bootstrap replayed 1/1 log segments. Stats: ops{read=5 overwritten=0 applied=5 ignored=0} inserts{seen=3 ignored=0} mutations{seen=0 ignored=0} orphaned_commits=0. Pending: 0 replicates
I20260929 02:00:10.392383 13331 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332: Bootstrap complete.
I20260929 02:00:10.402482 13331 raft_consensus.cc:359] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [term 1 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: true } }
I20260929 02:00:10.403393 13331 raft_consensus.cc:740] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [term 1 FOLLOWER]: Becoming Follower/Learner. State: Replica: b97759321664486dad92e9d63f3f5332, State: Initialized, Role: FOLLOWER
I20260929 02:00:10.404196 13331 consensus_queue.cc:260] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 5, Last appended: 1.5, Last appended by leader: 5, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: true } }
I20260929 02:00:10.405071 13337 sys_catalog.cc:455] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 committed_config { opid_index: 5 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: NON_VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: true } } }
I20260929 02:00:10.405388 13337 sys_catalog.cc:458] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:00:10.406248 13331 sys_catalog.cc:565] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:00:10.416951 13348 catalog_manager.cc:1308] Loaded cluster ID: 3145234aa8a04216b43cf8dd7e4989b8
I20260929 02:00:10.417058 13348 catalog_manager.cc:1601] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332: loading cluster ID for follower catalog manager: success
I20260929 02:00:10.424692 13348 catalog_manager.cc:1623] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332: acquiring CA information for follower catalog manager: success
I20260929 02:00:10.427330 13348 catalog_manager.cc:1651] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
I20260929 02:00:10.514166 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 13114
I20260929 02:00:10.532106 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:41917
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:42039,127.0.71.125:40905,127.0.71.124:33787
--webserver_port=33807
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:36581,127.0.71.85:45153,127.0.71.86:42537,127.0.71.87:34613,127.0.71.88:45991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:42039,127.0.71.125:40905,127.0.71.124:33787,127.0.71.123:40849 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:10.870573 13355 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:10.870985 13355 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:10.871146 13355 flags.cc:432] Enabled unsafe flag: --never_fsync=true
I20260929 02:00:10.873224 13306 raft_consensus.cc:2955] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [term 1 FOLLOWER]: Committing config change with OpId 1.6: config changed from index 5 to 6, 18262cb753d041a788c761bb04bb2a52 (127.0.71.123) changed from NON_VOTER to VOTER. New config: { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: false } } }
I20260929 02:00:10.880380 13361 sys_catalog.cc:455] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: false } } }
I20260929 02:00:10.880818 13361 sys_catalog.cc:458] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:00:10.882014 13337 sys_catalog.cc:455] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 0d3ea94ffef54f3fab0621aff4dab4df. Latest consensus state: current_term: 1 leader_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: false } } }
I20260929 02:00:10.882222 13337 sys_catalog.cc:458] T 00000000000000000000000000000000 P b97759321664486dad92e9d63f3f5332 [sys.catalog]: This master's current role is: FOLLOWER
W20260929 02:00:10.883697 13355 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:10.883920 13355 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 02:00:10.895421 13355 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:36581,127.0.71.85:45153,127.0.71.86:42537,127.0.71.87:34613,127.0.71.88:45991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:41917
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=33807
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:42039,127.0.71.125:40905,127.0.71.124:33787,127.0.71.123:40849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:10.897015 13355 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:10.898311 13355 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:10.907889 13364 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:10.907910 13363 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:10.908797 13366 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:10.909421 13355 server_base.cc:1061] running on GCE node
I20260929 02:00:10.910022 13355 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:10.911159 13355 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:10.914507 13355 hybrid_clock.cc:648] HybridClock initialized: now 1790647210914212 us; error 261 us; skew 500 ppm
I20260929 02:00:10.914836 13355 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:10 dist-test-slave-vz23 krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:10.918112 13355 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 02:00:10.919720 13355 webserver.cc:533] Webserver started at http://127.0.71.65:33807/ using document root <none> and password file <none>
I20260929 02:00:10.920507 13355 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:10.920621 13355 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:10.926198 13355 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.001s
I20260929 02:00:10.928937 13373 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:10.930366 13355 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:00:10.930533 13355 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "6ab6889a9df84d01bf9a0ebd21787d5f"
format_stamp: "Formatted at 2026-09-29 02:00:07 on dist-test-slave-vz23"
I20260929 02:00:10.931231 13355 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:10.965921 13355 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:10.974479 13355 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:10.974778 13355 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:10.975755 13355 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 02:00:10.977376 13355 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:00:10.977489 13355 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:10.977607 13355 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 02:00:10.977705 13355 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:11.015404 13355 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:41917
I20260929 02:00:11.015494 13485 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:41917 every 8 connection(s)
I20260929 02:00:11.016983 13355 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 02:00:11.019846 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 13355
Sep 29 02:00:11 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647188, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for hive/127.0.0.1@KRBTEST.COM
Sep 29 02:00:11 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.124@KRBTEST.COM
Sep 29 02:00:11 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.123@KRBTEST.COM
Sep 29 02:00:11 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.125@KRBTEST.COM
Sep 29 02:00:11 dist-test-slave-vz23 krb5kdc[12179](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647210, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 02:00:11.067272 13486 heartbeater.cc:344] Connected to a master server at 127.0.71.124:33787
I20260929 02:00:11.067698 13486 heartbeater.cc:461] Registering TS with master...
I20260929 02:00:11.069032 13486 heartbeater.cc:507] Master 127.0.71.124:33787 requested a full tablet report, sending...
I20260929 02:00:11.070604 13488 heartbeater.cc:344] Connected to a master server at 127.0.71.123:40849
I20260929 02:00:11.070776 13488 heartbeater.cc:461] Registering TS with master...
I20260929 02:00:11.071226 13488 heartbeater.cc:507] Master 127.0.71.123:40849 requested a full tablet report, sending...
I20260929 02:00:11.071426 12667 ts_manager.cc:194] Re-registered known tserver with Master: 6ab6889a9df84d01bf9a0ebd21787d5f (127.0.71.65:41917)
I20260929 02:00:11.074529 13061 ts_manager.cc:194] Re-registered known tserver with Master: 6ab6889a9df84d01bf9a0ebd21787d5f (127.0.71.65:41917)
I20260929 02:00:11.078675 13489 heartbeater.cc:344] Connected to a master server at 127.0.71.125:40905
I20260929 02:00:11.078843 13489 heartbeater.cc:461] Registering TS with master...
I20260929 02:00:11.078943 13487 heartbeater.cc:344] Connected to a master server at 127.0.71.126:42039
I20260929 02:00:11.079094 13487 heartbeater.cc:461] Registering TS with master...
I20260929 02:00:11.079493 13489 heartbeater.cc:507] Master 127.0.71.125:40905 requested a full tablet report, sending...
I20260929 02:00:11.079612 13487 heartbeater.cc:507] Master 127.0.71.126:42039 requested a full tablet report, sending...
I20260929 02:00:11.080711 12591 ts_manager.cc:194] Re-registered known tserver with Master: 6ab6889a9df84d01bf9a0ebd21787d5f (127.0.71.65:41917)
I20260929 02:00:11.081656 12591 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:60635
I20260929 02:00:11.081660 13296 ts_manager.cc:194] Registered new tserver with Master: 6ab6889a9df84d01bf9a0ebd21787d5f (127.0.71.65:41917)
I20260929 02:00:11.183107 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 13355
W20260929 02:00:11.184396 13503 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 02:00:11.180990 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:44344 (local address 127.0.71.65:41917)
0929 02:00:11.181766 (+ 776us) server_negotiation.cc:207] Beginning negotiation
0929 02:00:11.181774 (+ 8us) server_negotiation.cc:400] Waiting for connection header
0929 02:00:11.181804 (+ 30us) server_negotiation.cc:408] Connection header received
0929 02:00:11.181870 (+ 66us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 02:00:11.181877 (+ 7us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 02:00:11.182081 (+ 204us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 02:00:11.182248 (+ 167us) server_negotiation.cc:227] Negotiated authn=SASL
0929 02:00:11.182984 (+ 736us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:44344: BlockingRecv error: recv got EOF from 127.0.0.1:44344 (error 108)
Metrics: {"server-negotiator.queue_time_us":640,"thread_start_us":495,"threads_started":1}
W20260929 02:00:11.185361 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 13355: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:41917: server requires authentication, but client does not have Kerberos credentials available
I20260929 02:00:11.185489 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 13355
I20260929 02:00:11.204087 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:41917
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:42039,127.0.71.125:40905,127.0.71.124:33787
--webserver_port=33807
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:36581,127.0.71.85:45153,127.0.71.86:42537,127.0.71.87:34613,127.0.71.88:45991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:42039,127.0.71.125:40905,127.0.71.124:33787,127.0.71.123:40849
--fault_before_start=1.0 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
I20260929 02:00:11.484140 13256 sys_catalog.cc:455] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" committed_config { opid_index: 6 OBSOLETE_local: false peers { permanent_uuid: "b97759321664486dad92e9d63f3f5332" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 42039 } } peers { permanent_uuid: "0d3ea94ffef54f3fab0621aff4dab4df" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 40905 } } peers { permanent_uuid: "e9e01a8a110644a8918fce38a4c195d4" member_type: VOTER last_known_addr { host: "127.0.71.124" port: 33787 } } peers { permanent_uuid: "18262cb753d041a788c761bb04bb2a52" member_type: VOTER last_known_addr { host: "127.0.71.123" port: 40849 } attrs { promote: false } } }
I20260929 02:00:11.484426 13256 sys_catalog.cc:458] T 00000000000000000000000000000000 P 0d3ea94ffef54f3fab0621aff4dab4df [sys.catalog]: This master's current role is: LEADER
W20260929 02:00:11.561801 13505 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:11.562106 13505 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:11.562299 13505 flags.cc:432] Enabled unsafe flag: --fault_before_start=1
W20260929 02:00:11.562399 13505 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:11.571272 13505 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:11.571530 13505 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 02:00:11.582335 13505 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:36581,127.0.71.85:45153,127.0.71.86:42537,127.0.71.87:34613,127.0.71.88:45991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:41917
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=33807
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:42039,127.0.71.125:40905,127.0.71.124:33787,127.0.71.123:40849
--fault_before_start=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:11.584096 13505 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:11.585621 13505 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:11.595619 13514 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:11.595595 13513 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:11.596668 13516 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:11.597162 13505 server_base.cc:1061] running on GCE node
I20260929 02:00:11.597793 13505 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:11.598991 13505 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:11.600463 13505 hybrid_clock.cc:648] HybridClock initialized: now 1790647211600276 us; error 111 us; skew 500 ppm
I20260929 02:00:11.600767 13505 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:11 dist-test-slave-vz23 krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647211, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:11.603932 13505 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 02:00:11.605486 13505 webserver.cc:533] Webserver started at http://127.0.71.65:33807/ using document root <none> and password file <none>
I20260929 02:00:11.606209 13505 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:11.606321 13505 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:11.612219 13505 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.003s sys 0.000s
I20260929 02:00:11.615125 13523 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:11.616731 13505 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:00:11.616902 13505 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "6ab6889a9df84d01bf9a0ebd21787d5f"
format_stamp: "Formatted at 2026-09-29 02:00:07 on dist-test-slave-vz23"
I20260929 02:00:11.617600 13505 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:11.643036 13505 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:11.646955 13505 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:11.647182 13505 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:11.648053 13505 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 02:00:11.649659 13505 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:00:11.649732 13505 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:11.649812 13505 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 02:00:11.649856 13505 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
W20260929 02:00:11.650179 13505 fault_injection.cc:43] FAULT INJECTION ENABLED!
W20260929 02:00:11.650231 13505 fault_injection.cc:44] THIS SERVER MAY CRASH!
E20260929 02:00:11.650336 13505 fault_injection.cc:59] injecting fault for kudu (pid 13505): FLAGS_fault_before_start (process will exit)
I20260929 02:00:11.671624 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:41917
--local_ip_for_outbound_sockets=127.0.71.65
--tserver_master_addrs=127.0.71.126:42039,127.0.71.125:40905,127.0.71.124:33787
--webserver_port=33807
--webserver_interface=127.0.71.65
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:36581,127.0.71.85:45153,127.0.71.86:42537,127.0.71.87:34613,127.0.71.88:45991
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--tserver_master_addrs=127.0.71.126:42039,127.0.71.125:40905,127.0.71.124:33787,127.0.71.123:40849 with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:12.005550 13531 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:12.005874 13531 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:12.006044 13531 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:12.014699 13531 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:12.014947 13531 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 02:00:12.025933 13531 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:36581,127.0.71.85:45153,127.0.71.86:42537,127.0.71.87:34613,127.0.71.88:45991
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:41917
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=33807
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:42039,127.0.71.125:40905,127.0.71.124:33787,127.0.71.123:40849
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:12.027539 13531 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:12.028858 13531 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:12.038482 13539 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:12.038569 13538 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:12.039081 13541 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:12.040040 13531 server_base.cc:1061] running on GCE node
I20260929 02:00:12.040805 13531 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:12.041999 13531 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:12.043351 13531 hybrid_clock.cc:648] HybridClock initialized: now 1790647212043202 us; error 109 us; skew 500 ppm
I20260929 02:00:12.043700 13531 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:12 dist-test-slave-vz23 krb5kdc[12179](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647212, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:12.047470 13531 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 02:00:12.049312 13531 webserver.cc:533] Webserver started at http://127.0.71.65:33807/ using document root <none> and password file <none>
I20260929 02:00:12.050173 13531 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:12.050323 13531 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:12.055795 13531 fs_manager.cc:714] Time spent opening directory manager: real 0.004s user 0.002s sys 0.004s
I20260929 02:00:12.058729 13548 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:12.060096 13531 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 02:00:12.060242 13531 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "6ab6889a9df84d01bf9a0ebd21787d5f"
format_stamp: "Formatted at 2026-09-29 02:00:07 on dist-test-slave-vz23"
I20260929 02:00:12.061291 13531 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:12.087210 13531 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:12.091410 13531 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:12.091651 13531 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:12.092657 13531 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 02:00:12.094364 13531 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:00:12.094452 13531 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:12.094537 13531 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 02:00:12.094594 13531 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:12.130960 13531 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:41917
I20260929 02:00:12.131060 13660 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:41917 every 8 connection(s)
I20260929 02:00:12.132553 13531 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestAddMaster_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 02:00:12.137843 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 13531
I20260929 02:00:12.138240 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 13531
W20260929 02:00:12.166996 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 13531: Not authorized: Client connection negotiation failed: client connection to 127.0.71.65:41917: server requires authentication, but client does not have Kerberos credentials available
I20260929 02:00:12.167148 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 13531
I20260929 02:00:12.184639 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 13264
W20260929 02:00:12.185323 13533 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 02:00:12.137490 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:46749 (local address 127.0.71.126:42039)
0929 02:00:12.137695 (+ 205us) server_negotiation.cc:207] Beginning negotiation
0929 02:00:12.137702 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0929 02:00:12.161853 (+ 24151us) server_negotiation.cc:408] Connection header received
0929 02:00:12.162143 (+ 290us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 02:00:12.162153 (+ 10us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 02:00:12.162269 (+ 116us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 02:00:12.162353 (+ 84us) server_negotiation.cc:227] Negotiated authn=SASL
0929 02:00:12.184391 (+ 22038us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:46749: BlockingRecv error: recv got EOF from 127.0.71.65:46749 (error 108)
Metrics: {"server-negotiator.queue_time_us":61}
W20260929 02:00:12.185474 13668 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 02:00:12.137187 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:54857 (local address 127.0.71.125:40905)
0929 02:00:12.169564 (+ 32377us) server_negotiation.cc:207] Beginning negotiation
0929 02:00:12.169569 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0929 02:00:12.169583 (+ 14us) server_negotiation.cc:408] Connection header received
0929 02:00:12.169631 (+ 48us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 02:00:12.169635 (+ 4us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 02:00:12.169693 (+ 58us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 02:00:12.169785 (+ 92us) server_negotiation.cc:227] Negotiated authn=SASL
0929 02:00:12.184365 (+ 14580us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:54857: BlockingRecv error: recv error from unknown peer: Transport endpoint is not connected (error 107)
Metrics: {"server-negotiator.queue_time_us":32293,"thread_start_us":32199,"threads_started":1}
W20260929 02:00:12.185518 13666 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 02:00:12.136851 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:39915 (local address 127.0.71.124:33787)
0929 02:00:12.169951 (+ 33100us) server_negotiation.cc:207] Beginning negotiation
0929 02:00:12.169960 (+ 9us) server_negotiation.cc:400] Waiting for connection header
0929 02:00:12.184759 (+ 14799us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:39915: BlockingRecv error: recv got EOF from 127.0.71.65:39915 (error 108)
Metrics: {"server-negotiator.queue_time_us":32979,"thread_start_us":32850,"threads_started":1}
W20260929 02:00:12.186291 13672 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 02:00:12.148780 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.71.65:58095 (local address 127.0.71.123:40849)
0929 02:00:12.169263 (+ 20483us) server_negotiation.cc:207] Beginning negotiation
0929 02:00:12.169268 (+ 5us) server_negotiation.cc:400] Waiting for connection header
0929 02:00:12.185263 (+ 15995us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.71.65:58095: BlockingRecv error: recv got EOF from 127.0.71.65:58095 (error 108)
Metrics: {"server-negotiator.queue_time_us":20389,"thread_start_us":20275,"threads_started":1}
W20260929 02:00:12.187603 285 external_mini_cluster.cc:1721] Unable to check leaks on /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 13264: Not authorized: Client connection negotiation failed: client connection to 127.0.71.126:42039: server requires authentication, but client does not have Kerberos credentials available
W20260929 02:00:12.187650 13533 negotiation.cc:336] Failed RPC negotiation. Trace:
0929 02:00:12.186139 (+ 0us) reactor.cc:730] Submitting negotiation task for server connection from 127.0.0.1:51478 (local address 127.0.71.126:42039)
0929 02:00:12.186320 (+ 181us) server_negotiation.cc:207] Beginning negotiation
0929 02:00:12.186327 (+ 7us) server_negotiation.cc:400] Waiting for connection header
0929 02:00:12.186365 (+ 38us) server_negotiation.cc:408] Connection header received
0929 02:00:12.186574 (+ 209us) server_negotiation.cc:366] Received NEGOTIATE NegotiatePB request
0929 02:00:12.186580 (+ 6us) server_negotiation.cc:462] Received NEGOTIATE request from client
0929 02:00:12.186651 (+ 71us) server_negotiation.cc:378] Sending NEGOTIATE NegotiatePB response
0929 02:00:12.186753 (+ 102us) server_negotiation.cc:227] Negotiated authn=SASL
0929 02:00:12.187447 (+ 694us) negotiation.cc:326] Negotiation complete: Network error: Server connection negotiation failed: server connection from 127.0.0.1:51478: BlockingRecv error: recv got EOF from 127.0.0.1:51478 (error 108)
Metrics: {"server-negotiator.queue_time_us":49}
I20260929 02:00:12.187736 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 13264
I20260929 02:00:12.207742 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 12559
I20260929 02:00:12.208985 12621 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:00:12.326256 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 12559
W20260929 02:00:12.347134 12654 connection.cc:573] server connection from 127.0.0.1:52482 recv error: Network error: recv error from unknown peer: Transport endpoint is not connected (error 107)
I20260929 02:00:12.347388 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 12636
I20260929 02:00:12.348626 12698 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:00:12.463289 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 12636
I20260929 02:00:12.483588 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 12918
I20260929 02:00:12.484776 13091 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:00:12.634383 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 12918
Shutting down hive metastore.
2026-09-29T02:00:13Z chronyd exiting
2026-09-29T02:00:13Z chronyd exiting
2026-09-29T02:00:13Z chronyd exiting
2026-09-29T02:00:13Z chronyd exiting
2026-09-29T02:00:13Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestAddMaster/11 (26705 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/0
Loading random data
Initializing database '/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:00:13 dist-test-slave-vz23 krb5kdc[13684](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:00:13 dist-test-slave-vz23 krb5kdc[13684](info): set up 2 sockets
Sep 29 02:00:13 dist-test-slave-vz23 krb5kdc[13684](info): commencing operation
krb5kdc: starting...
W20260929 02:00:15.173259 285 mini_kdc.cc:121] Time spent starting KDC: real 2.052s user 0.004s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 02:00:15 dist-test-slave-vz23 krb5kdc[13684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:00:15Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:00:15Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:15.496408 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:45281
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:35481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:45281
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:15.829579 13700 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:15.829969 13700 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:15.830144 13700 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:15.839392 13700 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:00:15.839566 13700 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:15.839680 13700 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:00:15.839762 13700 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:00:15.850895 13700 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:45281
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:45281
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:15.852614 13700 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:15.853937 13700 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:15.861857 13706 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:15.862412 13705 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:15.863008 13708 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:15.863485 13700 server_base.cc:1061] running on GCE node
I20260929 02:00:15.864215 13700 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:15.865737 13700 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:15.866971 13700 hybrid_clock.cc:648] HybridClock initialized: now 1790647215866931 us; error 58 us; skew 500 ppm
I20260929 02:00:15.867362 13700 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:15 dist-test-slave-vz23 krb5kdc[13684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647215, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:15.870467 13700 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 02:00:15.872159 13700 webserver.cc:533] Webserver started at http://127.0.71.126:35779/ using document root <none> and password file <none>
I20260929 02:00:15.872956 13700 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:15.873087 13700 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:15.873426 13700 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:15.875820 13700 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "463908faa8754a4cac6cebd0f5ce939c"
format_stamp: "Formatted at 2026-09-29 02:00:15 on dist-test-slave-vz23"
I20260929 02:00:15.876643 13700 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "463908faa8754a4cac6cebd0f5ce939c"
format_stamp: "Formatted at 2026-09-29 02:00:15 on dist-test-slave-vz23"
I20260929 02:00:15.881919 13700 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:00:15.885218 13715 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:15.886507 13700 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:00:15.886798 13700 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "463908faa8754a4cac6cebd0f5ce939c"
format_stamp: "Formatted at 2026-09-29 02:00:15 on dist-test-slave-vz23"
I20260929 02:00:15.886999 13700 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:15.935683 13700 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:15.939903 13700 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:15.940210 13700 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:15.961021 13700 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:45281
I20260929 02:00:15.961046 13766 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:45281 every 8 connection(s)
I20260929 02:00:15.962509 13700 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 02:00:15.966600 13767 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:00:15.971807 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 13700
I20260929 02:00:15.972077 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 02:00:15.977085 13767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 463908faa8754a4cac6cebd0f5ce939c: Bootstrap starting.
I20260929 02:00:15.980949 13767 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 463908faa8754a4cac6cebd0f5ce939c: Neither blocks nor log segments found. Creating new log.
I20260929 02:00:15.982084 13767 log.cc:826] T 00000000000000000000000000000000 P 463908faa8754a4cac6cebd0f5ce939c: Log is configured to *not* fsync() on all Append() calls
I20260929 02:00:15.984911 13767 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 463908faa8754a4cac6cebd0f5ce939c: No bootstrap required, opened a new log
Sep 29 02:00:15 dist-test-slave-vz23 krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647215, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 02:00:15.990922 13767 raft_consensus.cc:359] T 00000000000000000000000000000000 P 463908faa8754a4cac6cebd0f5ce939c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463908faa8754a4cac6cebd0f5ce939c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 45281 } }
I20260929 02:00:15.991406 13767 raft_consensus.cc:385] T 00000000000000000000000000000000 P 463908faa8754a4cac6cebd0f5ce939c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:00:15.991530 13767 raft_consensus.cc:740] T 00000000000000000000000000000000 P 463908faa8754a4cac6cebd0f5ce939c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 463908faa8754a4cac6cebd0f5ce939c, State: Initialized, Role: FOLLOWER
I20260929 02:00:15.992419 13767 consensus_queue.cc:260] T 00000000000000000000000000000000 P 463908faa8754a4cac6cebd0f5ce939c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463908faa8754a4cac6cebd0f5ce939c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 45281 } }
I20260929 02:00:15.992753 13767 raft_consensus.cc:399] T 00000000000000000000000000000000 P 463908faa8754a4cac6cebd0f5ce939c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:00:15.992914 13767 raft_consensus.cc:493] T 00000000000000000000000000000000 P 463908faa8754a4cac6cebd0f5ce939c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:00:15.993098 13767 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 463908faa8754a4cac6cebd0f5ce939c [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:00:15.995493 13767 raft_consensus.cc:515] T 00000000000000000000000000000000 P 463908faa8754a4cac6cebd0f5ce939c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463908faa8754a4cac6cebd0f5ce939c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 45281 } }
I20260929 02:00:15.996043 13767 leader_election.cc:304] T 00000000000000000000000000000000 P 463908faa8754a4cac6cebd0f5ce939c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 463908faa8754a4cac6cebd0f5ce939c; no voters:
I20260929 02:00:15.996784 13767 leader_election.cc:290] T 00000000000000000000000000000000 P 463908faa8754a4cac6cebd0f5ce939c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:00:15.996946 13772 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 463908faa8754a4cac6cebd0f5ce939c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:00:15.998446 13772 raft_consensus.cc:697] T 00000000000000000000000000000000 P 463908faa8754a4cac6cebd0f5ce939c [term 1 LEADER]: Becoming Leader. State: Replica: 463908faa8754a4cac6cebd0f5ce939c, State: Running, Role: LEADER
I20260929 02:00:15.999027 13772 consensus_queue.cc:237] T 00000000000000000000000000000000 P 463908faa8754a4cac6cebd0f5ce939c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463908faa8754a4cac6cebd0f5ce939c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 45281 } }
I20260929 02:00:15.999157 13767 sys_catalog.cc:565] T 00000000000000000000000000000000 P 463908faa8754a4cac6cebd0f5ce939c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:00:16.001770 13774 sys_catalog.cc:455] T 00000000000000000000000000000000 P 463908faa8754a4cac6cebd0f5ce939c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 463908faa8754a4cac6cebd0f5ce939c. Latest consensus state: current_term: 1 leader_uuid: "463908faa8754a4cac6cebd0f5ce939c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463908faa8754a4cac6cebd0f5ce939c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 45281 } } }
I20260929 02:00:16.002480 13773 sys_catalog.cc:455] T 00000000000000000000000000000000 P 463908faa8754a4cac6cebd0f5ce939c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "463908faa8754a4cac6cebd0f5ce939c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "463908faa8754a4cac6cebd0f5ce939c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 45281 } } }
I20260929 02:00:16.002794 13773 sys_catalog.cc:458] T 00000000000000000000000000000000 P 463908faa8754a4cac6cebd0f5ce939c [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:16.002070 13774 sys_catalog.cc:458] T 00000000000000000000000000000000 P 463908faa8754a4cac6cebd0f5ce939c [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:16.009660 13778 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 02:00:16.016840 13778 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 02:00:16.026376 13778 catalog_manager.cc:1396] Generated new cluster ID: 5a28c20925d3410fbd1dc56419c931a3
I20260929 02:00:16.026505 13778 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 02:00:16.047260 13778 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 02:00:16.048770 13778 catalog_manager.cc:1553] Loading token signing keys...
I20260929 02:00:16.063340 13778 catalog_manager.cc:6219] T 00000000000000000000000000000000 P 463908faa8754a4cac6cebd0f5ce939c: Generated new TSK 0
I20260929 02:00:16.064381 13778 catalog_manager.cc:1563] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:16.186494 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:45281
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:35481
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:16.534621 13796 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:16.534970 13796 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:16.535121 13796 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:16.543678 13796 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:16.543923 13796 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 02:00:16.554560 13796 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35481
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:45281
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:16.556186 13796 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:16.557562 13796 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:16.567211 13801 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:16.567135 13802 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:16.567835 13804 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:16.568657 13796 server_base.cc:1061] running on GCE node
I20260929 02:00:16.569145 13796 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:16.570045 13796 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:16.571250 13796 hybrid_clock.cc:648] HybridClock initialized: now 1790647216571187 us; error 55 us; skew 500 ppm
I20260929 02:00:16.571563 13796 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:16 dist-test-slave-vz23 krb5kdc[13684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:16.574725 13796 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 02:00:16.576337 13796 webserver.cc:533] Webserver started at http://127.0.71.65:33585/ using document root <none> and password file <none>
I20260929 02:00:16.577081 13796 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:16.577216 13796 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:16.577541 13796 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:16.579838 13796 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "48e53000c2284a2e88a71bd9e77d538b"
format_stamp: "Formatted at 2026-09-29 02:00:16 on dist-test-slave-vz23"
I20260929 02:00:16.580673 13796 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "48e53000c2284a2e88a71bd9e77d538b"
format_stamp: "Formatted at 2026-09-29 02:00:16 on dist-test-slave-vz23"
I20260929 02:00:16.585815 13796 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.003s
I20260929 02:00:16.589200 13811 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:16.590521 13796 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:00:16.590691 13796 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "48e53000c2284a2e88a71bd9e77d538b"
format_stamp: "Formatted at 2026-09-29 02:00:16 on dist-test-slave-vz23"
I20260929 02:00:16.590891 13796 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:16.609072 13796 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:16.612635 13796 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:16.612922 13796 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:16.613840 13796 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 02:00:16.615338 13796 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:00:16.615446 13796 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:16.615592 13796 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 02:00:16.615664 13796 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:16.651305 13796 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:33675
I20260929 02:00:16.651371 13923 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:33675 every 8 connection(s)
I20260929 02:00:16.652726 13796 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 02:00:16.655289 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 13796
I20260929 02:00:16.655503 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_0.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
Sep 29 02:00:16 dist-test-slave-vz23 krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647216, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 02:00:16.672511 13924 heartbeater.cc:344] Connected to a master server at 127.0.71.126:45281
I20260929 02:00:16.672909 13924 heartbeater.cc:461] Registering TS with master...
I20260929 02:00:16.673679 13924 heartbeater.cc:507] Master 127.0.71.126:45281 requested a full tablet report, sending...
I20260929 02:00:16.675650 13732 ts_manager.cc:194] Registered new tserver with Master: 48e53000c2284a2e88a71bd9e77d538b (127.0.71.65:33675)
I20260929 02:00:16.677240 13732 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:49767
I20260929 02:00:16.683718 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Sep 29 02:00:16 dist-test-slave-vz23 krb5kdc[13684](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:00:16 dist-test-slave-vz23 krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.0.71.126@KRBTEST.COM
I20260929 02:00:16.737519 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 13796
Sep 29 02:00:16 dist-test-slave-vz23 krb5kdc[13684](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647216, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 02:00:16.754786 13919 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:00:16.841212 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 13796
I20260929 02:00:16.858048 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 13700
I20260929 02:00:16.859310 13762 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:00:16.965529 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 13700
2026-09-29T02:00:16Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/0 (3889 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/1
Loading random data
Initializing database '/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:00:17 dist-test-slave-vz23 krb5kdc[13937](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:00:17 dist-test-slave-vz23 krb5kdc[13937](info): set up 2 sockets
Sep 29 02:00:17 dist-test-slave-vz23 krb5kdc[13937](info): commencing operation
krb5kdc: starting...
W20260929 02:00:19.145097 285 mini_kdc.cc:121] Time spent starting KDC: real 2.135s user 0.007s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 02:00:19 dist-test-slave-vz23 krb5kdc[13937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:00:19Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:00:19Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:19.475555 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:35609
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:39079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:35609
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:19.807212 13953 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:19.807569 13953 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:19.807718 13953 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:19.816744 13953 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:00:19.816890 13953 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:19.817013 13953 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:00:19.817107 13953 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:00:19.828434 13953 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:39079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:35609
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:35609
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:19.830173 13953 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:19.831658 13953 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:19.840595 13958 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:19.840566 13959 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:19.841172 13961 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:19.842420 13953 server_base.cc:1061] running on GCE node
I20260929 02:00:19.843226 13953 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:19.844746 13953 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:19.846010 13953 hybrid_clock.cc:648] HybridClock initialized: now 1790647219845973 us; error 44 us; skew 500 ppm
I20260929 02:00:19.846490 13953 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:19 dist-test-slave-vz23 krb5kdc[13937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647219, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:19.849768 13953 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 02:00:19.851329 13953 webserver.cc:533] Webserver started at http://127.0.71.126:43103/ using document root <none> and password file <none>
I20260929 02:00:19.852051 13953 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:19.852200 13953 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:19.852629 13953 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:19.855005 13953 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "e166f690e54d4bd9bba972ef4aa3fdeb"
format_stamp: "Formatted at 2026-09-29 02:00:19 on dist-test-slave-vz23"
I20260929 02:00:19.855767 13953 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "e166f690e54d4bd9bba972ef4aa3fdeb"
format_stamp: "Formatted at 2026-09-29 02:00:19 on dist-test-slave-vz23"
I20260929 02:00:19.861378 13953 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.001s
I20260929 02:00:19.864661 13968 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:19.865949 13953 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:00:19.866145 13953 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "e166f690e54d4bd9bba972ef4aa3fdeb"
format_stamp: "Formatted at 2026-09-29 02:00:19 on dist-test-slave-vz23"
I20260929 02:00:19.866341 13953 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:19.883200 13953 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:19.886991 13953 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:19.887300 13953 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:19.906023 13953 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:35609
I20260929 02:00:19.906059 14019 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:35609 every 8 connection(s)
I20260929 02:00:19.907562 13953 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 02:00:19.910123 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 13953
I20260929 02:00:19.910321 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 02:00:19.912210 14020 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:00:19 dist-test-slave-vz23 krb5kdc[13937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647219, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 02:00:19.924306 14020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e166f690e54d4bd9bba972ef4aa3fdeb: Bootstrap starting.
I20260929 02:00:19.928493 14020 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P e166f690e54d4bd9bba972ef4aa3fdeb: Neither blocks nor log segments found. Creating new log.
I20260929 02:00:19.929733 14020 log.cc:826] T 00000000000000000000000000000000 P e166f690e54d4bd9bba972ef4aa3fdeb: Log is configured to *not* fsync() on all Append() calls
I20260929 02:00:19.932814 14020 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P e166f690e54d4bd9bba972ef4aa3fdeb: No bootstrap required, opened a new log
I20260929 02:00:19.938216 14020 raft_consensus.cc:359] T 00000000000000000000000000000000 P e166f690e54d4bd9bba972ef4aa3fdeb [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e166f690e54d4bd9bba972ef4aa3fdeb" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35609 } }
I20260929 02:00:19.938599 14020 raft_consensus.cc:385] T 00000000000000000000000000000000 P e166f690e54d4bd9bba972ef4aa3fdeb [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:00:19.938730 14020 raft_consensus.cc:740] T 00000000000000000000000000000000 P e166f690e54d4bd9bba972ef4aa3fdeb [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: e166f690e54d4bd9bba972ef4aa3fdeb, State: Initialized, Role: FOLLOWER
I20260929 02:00:19.939359 14020 consensus_queue.cc:260] T 00000000000000000000000000000000 P e166f690e54d4bd9bba972ef4aa3fdeb [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e166f690e54d4bd9bba972ef4aa3fdeb" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35609 } }
I20260929 02:00:19.939564 14020 raft_consensus.cc:399] T 00000000000000000000000000000000 P e166f690e54d4bd9bba972ef4aa3fdeb [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:00:19.939700 14020 raft_consensus.cc:493] T 00000000000000000000000000000000 P e166f690e54d4bd9bba972ef4aa3fdeb [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:00:19.939875 14020 raft_consensus.cc:3060] T 00000000000000000000000000000000 P e166f690e54d4bd9bba972ef4aa3fdeb [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:00:19.942040 14020 raft_consensus.cc:515] T 00000000000000000000000000000000 P e166f690e54d4bd9bba972ef4aa3fdeb [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e166f690e54d4bd9bba972ef4aa3fdeb" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35609 } }
I20260929 02:00:19.942528 14020 leader_election.cc:304] T 00000000000000000000000000000000 P e166f690e54d4bd9bba972ef4aa3fdeb [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: e166f690e54d4bd9bba972ef4aa3fdeb; no voters:
I20260929 02:00:19.943181 14020 leader_election.cc:290] T 00000000000000000000000000000000 P e166f690e54d4bd9bba972ef4aa3fdeb [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:00:19.943334 14025 raft_consensus.cc:2804] T 00000000000000000000000000000000 P e166f690e54d4bd9bba972ef4aa3fdeb [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:00:19.944612 14025 raft_consensus.cc:697] T 00000000000000000000000000000000 P e166f690e54d4bd9bba972ef4aa3fdeb [term 1 LEADER]: Becoming Leader. State: Replica: e166f690e54d4bd9bba972ef4aa3fdeb, State: Running, Role: LEADER
I20260929 02:00:19.945111 14025 consensus_queue.cc:237] T 00000000000000000000000000000000 P e166f690e54d4bd9bba972ef4aa3fdeb [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e166f690e54d4bd9bba972ef4aa3fdeb" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35609 } }
I20260929 02:00:19.945413 14020 sys_catalog.cc:565] T 00000000000000000000000000000000 P e166f690e54d4bd9bba972ef4aa3fdeb [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:00:19.947906 14026 sys_catalog.cc:455] T 00000000000000000000000000000000 P e166f690e54d4bd9bba972ef4aa3fdeb [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "e166f690e54d4bd9bba972ef4aa3fdeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e166f690e54d4bd9bba972ef4aa3fdeb" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35609 } } }
I20260929 02:00:19.948226 14026 sys_catalog.cc:458] T 00000000000000000000000000000000 P e166f690e54d4bd9bba972ef4aa3fdeb [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:19.948098 14027 sys_catalog.cc:455] T 00000000000000000000000000000000 P e166f690e54d4bd9bba972ef4aa3fdeb [sys.catalog]: SysCatalogTable state changed. Reason: New leader e166f690e54d4bd9bba972ef4aa3fdeb. Latest consensus state: current_term: 1 leader_uuid: "e166f690e54d4bd9bba972ef4aa3fdeb" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "e166f690e54d4bd9bba972ef4aa3fdeb" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35609 } } }
I20260929 02:00:19.948341 14027 sys_catalog.cc:458] T 00000000000000000000000000000000 P e166f690e54d4bd9bba972ef4aa3fdeb [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:19.949997 14034 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 02:00:19.954583 14034 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 02:00:19.962765 14034 catalog_manager.cc:1396] Generated new cluster ID: 8cc468bcc702463e9ec1509f29277ba9
I20260929 02:00:19.962941 14034 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 02:00:19.975133 14034 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 02:00:19.976836 14034 catalog_manager.cc:1553] Loading token signing keys...
I20260929 02:00:20.000934 14034 catalog_manager.cc:6219] T 00000000000000000000000000000000 P e166f690e54d4bd9bba972ef4aa3fdeb: Generated new TSK 0
I20260929 02:00:20.002148 14034 catalog_manager.cc:1563] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:20.128638 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:35609
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:39079
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:20.483718 14048 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:20.484045 14048 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:20.484157 14048 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:20.493499 14048 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:20.493693 14048 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 02:00:20.504688 14048 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:39079
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:35609
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:20.506218 14048 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:20.507449 14048 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:20.516805 14053 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:20.516856 14054 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:20.517158 14056 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:20.518127 14048 server_base.cc:1061] running on GCE node
I20260929 02:00:20.518707 14048 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:20.519603 14048 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:20.520889 14048 hybrid_clock.cc:648] HybridClock initialized: now 1790647220520821 us; error 52 us; skew 500 ppm
I20260929 02:00:20.521232 14048 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:20 dist-test-slave-vz23 krb5kdc[13937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:20.524340 14048 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 02:00:20.525974 14048 webserver.cc:533] Webserver started at http://127.0.71.65:34603/ using document root <none> and password file <none>
I20260929 02:00:20.526649 14048 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:20.526739 14048 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:20.527014 14048 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:20.529451 14048 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "7ae6b53fbba54d12a2a1cc801fc9207b"
format_stamp: "Formatted at 2026-09-29 02:00:20 on dist-test-slave-vz23"
I20260929 02:00:20.530164 14048 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "7ae6b53fbba54d12a2a1cc801fc9207b"
format_stamp: "Formatted at 2026-09-29 02:00:20 on dist-test-slave-vz23"
I20260929 02:00:20.535389 14048 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 02:00:20.538611 14063 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:20.539975 14048 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.000s sys 0.002s
I20260929 02:00:20.540169 14048 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "7ae6b53fbba54d12a2a1cc801fc9207b"
format_stamp: "Formatted at 2026-09-29 02:00:20 on dist-test-slave-vz23"
I20260929 02:00:20.540385 14048 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:20.563654 14048 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:20.567713 14048 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:20.568017 14048 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:20.568993 14048 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 02:00:20.570627 14048 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:00:20.570719 14048 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:20.570895 14048 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 02:00:20.570961 14048 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:20.605059 14048 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:43999
I20260929 02:00:20.605187 14175 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:43999 every 8 connection(s)
I20260929 02:00:20.606508 14048 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 02:00:20.615479 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 14048
I20260929 02:00:20.615650 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_1.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
Sep 29 02:00:20 dist-test-slave-vz23 krb5kdc[13937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647220, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 02:00:20.626124 14176 heartbeater.cc:344] Connected to a master server at 127.0.71.126:35609
I20260929 02:00:20.626518 14176 heartbeater.cc:461] Registering TS with master...
I20260929 02:00:20.627348 14176 heartbeater.cc:507] Master 127.0.71.126:35609 requested a full tablet report, sending...
I20260929 02:00:20.629662 13985 ts_manager.cc:194] Registered new tserver with Master: 7ae6b53fbba54d12a2a1cc801fc9207b (127.0.71.65:43999)
I20260929 02:00:20.631902 13985 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:36419
I20260929 02:00:20.632081 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Sep 29 02:00:20 dist-test-slave-vz23 krb5kdc[13937](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:00:20 dist-test-slave-vz23 krb5kdc[13937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.0.71.126@KRBTEST.COM
I20260929 02:00:20.686096 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 14048
Sep 29 02:00:20 dist-test-slave-vz23 krb5kdc[13937](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647220, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 02:00:20.702773 14171 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:00:20.791064 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 14048
I20260929 02:00:20.808104 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 13953
I20260929 02:00:20.809347 14015 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:00:20.915853 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 13953
2026-09-29T02:00:20Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/1 (3950 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/2
Loading random data
Initializing database '/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:00:21 dist-test-slave-vz23 krb5kdc[14189](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:00:21 dist-test-slave-vz23 krb5kdc[14189](info): set up 2 sockets
Sep 29 02:00:21 dist-test-slave-vz23 krb5kdc[14189](info): commencing operation
krb5kdc: starting...
W20260929 02:00:23.013574 285 mini_kdc.cc:121] Time spent starting KDC: real 2.055s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 02:00:23 dist-test-slave-vz23 krb5kdc[14189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:00:23Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:00:23Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:23.332914 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:34077
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:34077
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:23.671104 14205 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:23.671451 14205 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:23.671602 14205 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:23.679986 14205 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:00:23.680104 14205 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:23.680234 14205 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:00:23.680320 14205 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:00:23.691118 14205 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:34077
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:34077
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:23.692790 14205 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:23.694145 14205 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:23.702350 14210 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:23.702332 14211 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:23.702979 14213 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:23.704001 14205 server_base.cc:1061] running on GCE node
I20260929 02:00:23.705139 14205 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:23.706564 14205 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:23.707801 14205 hybrid_clock.cc:648] HybridClock initialized: now 1790647223707746 us; error 63 us; skew 500 ppm
I20260929 02:00:23.708148 14205 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:23 dist-test-slave-vz23 krb5kdc[14189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647223, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:23.711169 14205 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 02:00:23.712899 14205 webserver.cc:533] Webserver started at http://127.0.71.126:42985/ using document root <none> and password file <none>
I20260929 02:00:23.713606 14205 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:23.713735 14205 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:23.714135 14205 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:23.716686 14205 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "d067b16b87bc453dbaa883d47449e8e8"
format_stamp: "Formatted at 2026-09-29 02:00:23 on dist-test-slave-vz23"
I20260929 02:00:23.717456 14205 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "d067b16b87bc453dbaa883d47449e8e8"
format_stamp: "Formatted at 2026-09-29 02:00:23 on dist-test-slave-vz23"
I20260929 02:00:23.722760 14205 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:00:23.726145 14220 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:23.727451 14205 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:00:23.727648 14205 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "d067b16b87bc453dbaa883d47449e8e8"
format_stamp: "Formatted at 2026-09-29 02:00:23 on dist-test-slave-vz23"
I20260929 02:00:23.727850 14205 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:23.744647 14205 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:23.748265 14205 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:23.748661 14205 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:23.769373 14205 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:34077
I20260929 02:00:23.769457 14271 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:34077 every 8 connection(s)
I20260929 02:00:23.770884 14205 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 02:00:23.774964 14272 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:00:23.778311 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 14205
I20260929 02:00:23.778522 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 02:00:23.786922 14272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d067b16b87bc453dbaa883d47449e8e8: Bootstrap starting.
I20260929 02:00:23.791010 14272 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P d067b16b87bc453dbaa883d47449e8e8: Neither blocks nor log segments found. Creating new log.
I20260929 02:00:23.792084 14272 log.cc:826] T 00000000000000000000000000000000 P d067b16b87bc453dbaa883d47449e8e8: Log is configured to *not* fsync() on all Append() calls
Sep 29 02:00:23 dist-test-slave-vz23 krb5kdc[14189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647223, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 02:00:23.795245 14272 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P d067b16b87bc453dbaa883d47449e8e8: No bootstrap required, opened a new log
I20260929 02:00:23.800804 14272 raft_consensus.cc:359] T 00000000000000000000000000000000 P d067b16b87bc453dbaa883d47449e8e8 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d067b16b87bc453dbaa883d47449e8e8" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34077 } }
I20260929 02:00:23.801178 14272 raft_consensus.cc:385] T 00000000000000000000000000000000 P d067b16b87bc453dbaa883d47449e8e8 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:00:23.801303 14272 raft_consensus.cc:740] T 00000000000000000000000000000000 P d067b16b87bc453dbaa883d47449e8e8 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d067b16b87bc453dbaa883d47449e8e8, State: Initialized, Role: FOLLOWER
I20260929 02:00:23.801875 14272 consensus_queue.cc:260] T 00000000000000000000000000000000 P d067b16b87bc453dbaa883d47449e8e8 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d067b16b87bc453dbaa883d47449e8e8" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34077 } }
I20260929 02:00:23.802101 14272 raft_consensus.cc:399] T 00000000000000000000000000000000 P d067b16b87bc453dbaa883d47449e8e8 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:00:23.802239 14272 raft_consensus.cc:493] T 00000000000000000000000000000000 P d067b16b87bc453dbaa883d47449e8e8 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:00:23.802410 14272 raft_consensus.cc:3060] T 00000000000000000000000000000000 P d067b16b87bc453dbaa883d47449e8e8 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:00:23.804978 14272 raft_consensus.cc:515] T 00000000000000000000000000000000 P d067b16b87bc453dbaa883d47449e8e8 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d067b16b87bc453dbaa883d47449e8e8" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34077 } }
I20260929 02:00:23.805557 14272 leader_election.cc:304] T 00000000000000000000000000000000 P d067b16b87bc453dbaa883d47449e8e8 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: d067b16b87bc453dbaa883d47449e8e8; no voters:
I20260929 02:00:23.806206 14272 leader_election.cc:290] T 00000000000000000000000000000000 P d067b16b87bc453dbaa883d47449e8e8 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:00:23.806375 14277 raft_consensus.cc:2804] T 00000000000000000000000000000000 P d067b16b87bc453dbaa883d47449e8e8 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:00:23.807118 14277 raft_consensus.cc:697] T 00000000000000000000000000000000 P d067b16b87bc453dbaa883d47449e8e8 [term 1 LEADER]: Becoming Leader. State: Replica: d067b16b87bc453dbaa883d47449e8e8, State: Running, Role: LEADER
I20260929 02:00:23.807788 14277 consensus_queue.cc:237] T 00000000000000000000000000000000 P d067b16b87bc453dbaa883d47449e8e8 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d067b16b87bc453dbaa883d47449e8e8" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34077 } }
I20260929 02:00:23.808612 14272 sys_catalog.cc:565] T 00000000000000000000000000000000 P d067b16b87bc453dbaa883d47449e8e8 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:00:23.810683 14278 sys_catalog.cc:455] T 00000000000000000000000000000000 P d067b16b87bc453dbaa883d47449e8e8 [sys.catalog]: SysCatalogTable state changed. Reason: New leader d067b16b87bc453dbaa883d47449e8e8. Latest consensus state: current_term: 1 leader_uuid: "d067b16b87bc453dbaa883d47449e8e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d067b16b87bc453dbaa883d47449e8e8" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34077 } } }
I20260929 02:00:23.811065 14278 sys_catalog.cc:458] T 00000000000000000000000000000000 P d067b16b87bc453dbaa883d47449e8e8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:23.811594 14279 sys_catalog.cc:455] T 00000000000000000000000000000000 P d067b16b87bc453dbaa883d47449e8e8 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "d067b16b87bc453dbaa883d47449e8e8" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "d067b16b87bc453dbaa883d47449e8e8" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 34077 } } }
I20260929 02:00:23.812016 14279 sys_catalog.cc:458] T 00000000000000000000000000000000 P d067b16b87bc453dbaa883d47449e8e8 [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:23.815953 14284 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 02:00:23.821911 14284 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 02:00:23.830888 14284 catalog_manager.cc:1396] Generated new cluster ID: 70641af4f7a443c6b2b56a98ed2a26f3
I20260929 02:00:23.831028 14284 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 02:00:23.850893 14284 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 02:00:23.852746 14284 catalog_manager.cc:1553] Loading token signing keys...
I20260929 02:00:23.864964 14284 catalog_manager.cc:6219] T 00000000000000000000000000000000 P d067b16b87bc453dbaa883d47449e8e8: Generated new TSK 0
I20260929 02:00:23.866050 14284 catalog_manager.cc:1563] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:23.985950 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:34077
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:45121
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:24.329742 14300 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:24.330081 14300 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:24.330246 14300 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:24.338943 14300 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:24.339198 14300 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 02:00:24.349957 14300 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:45121
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:34077
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:24.351698 14300 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:24.353081 14300 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:24.363319 14305 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:24.363613 14306 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:24.367436 14300 server_base.cc:1061] running on GCE node
W20260929 02:00:24.368332 14308 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:24.368994 14300 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:24.369997 14300 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:24.371240 14300 hybrid_clock.cc:648] HybridClock initialized: now 1790647224371237 us; error 79 us; skew 500 ppm
I20260929 02:00:24.371665 14300 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:24 dist-test-slave-vz23 krb5kdc[14189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:24.374845 14300 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 02:00:24.376734 14300 webserver.cc:533] Webserver started at http://127.0.71.65:39107/ using document root <none> and password file <none>
I20260929 02:00:24.377414 14300 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:24.377506 14300 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:24.377800 14300 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:24.380053 14300 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "922297f82547456ba4eb60c5cd89970b"
format_stamp: "Formatted at 2026-09-29 02:00:24 on dist-test-slave-vz23"
I20260929 02:00:24.380837 14300 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "922297f82547456ba4eb60c5cd89970b"
format_stamp: "Formatted at 2026-09-29 02:00:24 on dist-test-slave-vz23"
I20260929 02:00:24.386121 14300 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.004s sys 0.000s
I20260929 02:00:24.389642 14315 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:24.391142 14300 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.001s
I20260929 02:00:24.391359 14300 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "922297f82547456ba4eb60c5cd89970b"
format_stamp: "Formatted at 2026-09-29 02:00:24 on dist-test-slave-vz23"
I20260929 02:00:24.391521 14300 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:24.409051 14300 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:24.412907 14300 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:24.413146 14300 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:24.414013 14300 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 02:00:24.415503 14300 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:00:24.415575 14300 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:24.415658 14300 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 02:00:24.415702 14300 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:24.450634 14300 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:38481
I20260929 02:00:24.450704 14427 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:38481 every 8 connection(s)
I20260929 02:00:24.452035 14300 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 02:00:24.453197 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 14300
I20260929 02:00:24.453375 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_2.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
Sep 29 02:00:24 dist-test-slave-vz23 krb5kdc[14189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647224, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 02:00:24.472077 14428 heartbeater.cc:344] Connected to a master server at 127.0.71.126:34077
I20260929 02:00:24.472435 14428 heartbeater.cc:461] Registering TS with master...
I20260929 02:00:24.473331 14428 heartbeater.cc:507] Master 127.0.71.126:34077 requested a full tablet report, sending...
I20260929 02:00:24.475723 14237 ts_manager.cc:194] Registered new tserver with Master: 922297f82547456ba4eb60c5cd89970b (127.0.71.65:38481)
I20260929 02:00:24.478071 14237 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:50421
I20260929 02:00:24.481688 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Sep 29 02:00:24 dist-test-slave-vz23 krb5kdc[14189](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:00:24 dist-test-slave-vz23 krb5kdc[14189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.0.71.126@KRBTEST.COM
I20260929 02:00:24.534360 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 14300
Sep 29 02:00:24 dist-test-slave-vz23 krb5kdc[14189](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647224, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 02:00:24.550537 14423 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:00:24.635424 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 14300
I20260929 02:00:24.652594 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 14205
I20260929 02:00:24.653801 14267 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:00:24.760318 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 14205
2026-09-29T02:00:24Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/2 (3844 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/3
Loading random data
Initializing database '/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:00:24 dist-test-slave-vz23 krb5kdc[14441](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:00:24 dist-test-slave-vz23 krb5kdc[14441](info): set up 2 sockets
Sep 29 02:00:24 dist-test-slave-vz23 krb5kdc[14441](info): commencing operation
krb5kdc: starting...
W20260929 02:00:26.857117 285 mini_kdc.cc:121] Time spent starting KDC: real 2.055s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 02:00:27 dist-test-slave-vz23 krb5kdc[14441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:00:27Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:00:27Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:27.190068 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:41439
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:41337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:41439
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:27.527964 14457 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:27.528296 14457 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:27.528383 14457 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:27.536973 14457 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:00:27.537091 14457 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:27.537155 14457 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:00:27.537204 14457 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:00:27.548264 14457 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:41337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:41439
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:41439
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:27.549880 14457 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:27.551239 14457 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:27.559098 14463 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:27.559093 14462 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:27.559994 14465 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:27.560743 14457 server_base.cc:1061] running on GCE node
I20260929 02:00:27.561472 14457 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:27.562829 14457 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:27.564106 14457 hybrid_clock.cc:648] HybridClock initialized: now 1790647227564050 us; error 38 us; skew 500 ppm
I20260929 02:00:27.564548 14457 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:27 dist-test-slave-vz23 krb5kdc[14441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647227, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:27.567937 14457 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 02:00:27.569793 14457 webserver.cc:533] Webserver started at http://127.0.71.126:33011/ using document root <none> and password file <none>
I20260929 02:00:27.570619 14457 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:27.570762 14457 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:27.571121 14457 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:27.573627 14457 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "c1f6fcac11464ba193594b77d2eddd04"
format_stamp: "Formatted at 2026-09-29 02:00:27 on dist-test-slave-vz23"
I20260929 02:00:27.574399 14457 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "c1f6fcac11464ba193594b77d2eddd04"
format_stamp: "Formatted at 2026-09-29 02:00:27 on dist-test-slave-vz23"
I20260929 02:00:27.579877 14457 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.001s
I20260929 02:00:27.583384 14472 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:27.584852 14457 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 02:00:27.585070 14457 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "c1f6fcac11464ba193594b77d2eddd04"
format_stamp: "Formatted at 2026-09-29 02:00:27 on dist-test-slave-vz23"
I20260929 02:00:27.585363 14457 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:27.607369 14457 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:27.618741 14457 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:27.619083 14457 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:27.641650 14457 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:41439
I20260929 02:00:27.641685 14523 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:41439 every 8 connection(s)
I20260929 02:00:27.643388 14457 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 02:00:27.645864 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 14457
I20260929 02:00:27.646051 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 02:00:27.648581 14524 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:00:27 dist-test-slave-vz23 krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647227, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 02:00:27.661640 14524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1f6fcac11464ba193594b77d2eddd04: Bootstrap starting.
I20260929 02:00:27.666246 14524 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P c1f6fcac11464ba193594b77d2eddd04: Neither blocks nor log segments found. Creating new log.
I20260929 02:00:27.667500 14524 log.cc:826] T 00000000000000000000000000000000 P c1f6fcac11464ba193594b77d2eddd04: Log is configured to *not* fsync() on all Append() calls
I20260929 02:00:27.670645 14524 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P c1f6fcac11464ba193594b77d2eddd04: No bootstrap required, opened a new log
I20260929 02:00:27.675863 14524 raft_consensus.cc:359] T 00000000000000000000000000000000 P c1f6fcac11464ba193594b77d2eddd04 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f6fcac11464ba193594b77d2eddd04" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41439 } }
I20260929 02:00:27.676208 14524 raft_consensus.cc:385] T 00000000000000000000000000000000 P c1f6fcac11464ba193594b77d2eddd04 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:00:27.676317 14524 raft_consensus.cc:740] T 00000000000000000000000000000000 P c1f6fcac11464ba193594b77d2eddd04 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: c1f6fcac11464ba193594b77d2eddd04, State: Initialized, Role: FOLLOWER
I20260929 02:00:27.677017 14524 consensus_queue.cc:260] T 00000000000000000000000000000000 P c1f6fcac11464ba193594b77d2eddd04 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f6fcac11464ba193594b77d2eddd04" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41439 } }
I20260929 02:00:27.677238 14524 raft_consensus.cc:399] T 00000000000000000000000000000000 P c1f6fcac11464ba193594b77d2eddd04 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:00:27.677384 14524 raft_consensus.cc:493] T 00000000000000000000000000000000 P c1f6fcac11464ba193594b77d2eddd04 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:00:27.677558 14524 raft_consensus.cc:3060] T 00000000000000000000000000000000 P c1f6fcac11464ba193594b77d2eddd04 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:00:27.679811 14524 raft_consensus.cc:515] T 00000000000000000000000000000000 P c1f6fcac11464ba193594b77d2eddd04 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f6fcac11464ba193594b77d2eddd04" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41439 } }
I20260929 02:00:27.680282 14524 leader_election.cc:304] T 00000000000000000000000000000000 P c1f6fcac11464ba193594b77d2eddd04 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: c1f6fcac11464ba193594b77d2eddd04; no voters:
I20260929 02:00:27.680974 14524 leader_election.cc:290] T 00000000000000000000000000000000 P c1f6fcac11464ba193594b77d2eddd04 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:00:27.681135 14529 raft_consensus.cc:2804] T 00000000000000000000000000000000 P c1f6fcac11464ba193594b77d2eddd04 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:00:27.682349 14529 raft_consensus.cc:697] T 00000000000000000000000000000000 P c1f6fcac11464ba193594b77d2eddd04 [term 1 LEADER]: Becoming Leader. State: Replica: c1f6fcac11464ba193594b77d2eddd04, State: Running, Role: LEADER
I20260929 02:00:27.682898 14529 consensus_queue.cc:237] T 00000000000000000000000000000000 P c1f6fcac11464ba193594b77d2eddd04 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f6fcac11464ba193594b77d2eddd04" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41439 } }
I20260929 02:00:27.683187 14524 sys_catalog.cc:565] T 00000000000000000000000000000000 P c1f6fcac11464ba193594b77d2eddd04 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:00:27.685868 14531 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1f6fcac11464ba193594b77d2eddd04 [sys.catalog]: SysCatalogTable state changed. Reason: New leader c1f6fcac11464ba193594b77d2eddd04. Latest consensus state: current_term: 1 leader_uuid: "c1f6fcac11464ba193594b77d2eddd04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f6fcac11464ba193594b77d2eddd04" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41439 } } }
I20260929 02:00:27.686175 14531 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1f6fcac11464ba193594b77d2eddd04 [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:27.685940 14530 sys_catalog.cc:455] T 00000000000000000000000000000000 P c1f6fcac11464ba193594b77d2eddd04 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "c1f6fcac11464ba193594b77d2eddd04" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "c1f6fcac11464ba193594b77d2eddd04" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 41439 } } }
I20260929 02:00:27.686343 14530 sys_catalog.cc:458] T 00000000000000000000000000000000 P c1f6fcac11464ba193594b77d2eddd04 [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:27.697897 14536 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 02:00:27.702857 14536 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 02:00:27.711797 14536 catalog_manager.cc:1396] Generated new cluster ID: 9695975b80604839a8dc9bb2d3e86bc9
I20260929 02:00:27.711942 14536 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 02:00:27.721395 14536 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 02:00:27.722596 14536 catalog_manager.cc:1553] Loading token signing keys...
I20260929 02:00:27.730041 14536 catalog_manager.cc:6219] T 00000000000000000000000000000000 P c1f6fcac11464ba193594b77d2eddd04: Generated new TSK 0
I20260929 02:00:27.730937 14536 catalog_manager.cc:1563] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:27.852785 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:41439
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:41337
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:28.195634 14552 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:28.195951 14552 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:28.196111 14552 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:28.204735 14552 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:28.204965 14552 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 02:00:28.215489 14552 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:41337
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:41439
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:28.217151 14552 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:28.218590 14552 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:28.227844 14557 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:28.227895 14558 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:28.228992 14552 server_base.cc:1061] running on GCE node
W20260929 02:00:28.229209 14560 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:28.229777 14552 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:28.230687 14552 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:28.231935 14552 hybrid_clock.cc:648] HybridClock initialized: now 1790647228231883 us; error 55 us; skew 500 ppm
I20260929 02:00:28.232250 14552 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:28 dist-test-slave-vz23 krb5kdc[14441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:28.235282 14552 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 02:00:28.236881 14552 webserver.cc:533] Webserver started at http://127.0.71.65:46175/ using document root <none> and password file <none>
I20260929 02:00:28.237582 14552 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:28.237715 14552 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:28.238052 14552 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:28.240373 14552 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "a802924535fe4ddb81a20d12e57dc1be"
format_stamp: "Formatted at 2026-09-29 02:00:28 on dist-test-slave-vz23"
I20260929 02:00:28.241209 14552 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "a802924535fe4ddb81a20d12e57dc1be"
format_stamp: "Formatted at 2026-09-29 02:00:28 on dist-test-slave-vz23"
I20260929 02:00:28.246482 14552 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.004s
I20260929 02:00:28.249776 14567 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:28.251067 14552 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:00:28.251250 14552 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "a802924535fe4ddb81a20d12e57dc1be"
format_stamp: "Formatted at 2026-09-29 02:00:28 on dist-test-slave-vz23"
I20260929 02:00:28.251473 14552 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:28.264899 14552 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:28.268428 14552 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:28.268842 14552 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:28.269809 14552 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 02:00:28.271366 14552 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:00:28.271469 14552 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:28.271622 14552 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 02:00:28.271697 14552 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:28.306905 14552 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:43623
I20260929 02:00:28.306964 14679 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:43623 every 8 connection(s)
I20260929 02:00:28.308350 14552 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 02:00:28.313269 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 14552
I20260929 02:00:28.313460 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_3.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
Sep 29 02:00:28 dist-test-slave-vz23 krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647228, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 02:00:28.329063 14680 heartbeater.cc:344] Connected to a master server at 127.0.71.126:41439
I20260929 02:00:28.329434 14680 heartbeater.cc:461] Registering TS with master...
I20260929 02:00:28.330231 14680 heartbeater.cc:507] Master 127.0.71.126:41439 requested a full tablet report, sending...
I20260929 02:00:28.332261 14489 ts_manager.cc:194] Registered new tserver with Master: a802924535fe4ddb81a20d12e57dc1be (127.0.71.65:43623)
I20260929 02:00:28.333822 14489 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:54369
I20260929 02:00:28.341190 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Sep 29 02:00:28 dist-test-slave-vz23 krb5kdc[14441](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:00:28 dist-test-slave-vz23 krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.0.71.126@KRBTEST.COM
I20260929 02:00:28.394848 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 14552
Sep 29 02:00:28 dist-test-slave-vz23 krb5kdc[14441](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647228, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 02:00:28.412755 14675 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:00:28.505416 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 14552
I20260929 02:00:28.522471 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 14457
I20260929 02:00:28.523578 14519 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:00:28.629103 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 14457
2026-09-29T02:00:28Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/3 (3885 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/4
Loading random data
Initializing database '/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:00:28 dist-test-slave-vz23 krb5kdc[14693](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:00:28 dist-test-slave-vz23 krb5kdc[14693](info): set up 2 sockets
Sep 29 02:00:28 dist-test-slave-vz23 krb5kdc[14693](info): commencing operation
krb5kdc: starting...
W20260929 02:00:30.737232 285 mini_kdc.cc:121] Time spent starting KDC: real 2.050s user 0.008s sys 0.023s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 02:00:30 dist-test-slave-vz23 krb5kdc[14693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:00:30Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:00:30Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:31.063930 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:44141
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:42321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:44141
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:31.402374 14709 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:31.402871 14709 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:31.403071 14709 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:31.414693 14709 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:00:31.414855 14709 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:31.414947 14709 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:00:31.415038 14709 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:00:31.425946 14709 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:42321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:44141
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:44141
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:31.427568 14709 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:31.428957 14709 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:31.437228 14715 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:31.437461 14714 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:31.437847 14717 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:31.438521 14709 server_base.cc:1061] running on GCE node
I20260929 02:00:31.439224 14709 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:31.440672 14709 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:31.441922 14709 hybrid_clock.cc:648] HybridClock initialized: now 1790647231441879 us; error 55 us; skew 500 ppm
I20260929 02:00:31.442322 14709 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:31 dist-test-slave-vz23 krb5kdc[14693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647231, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:31.445451 14709 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 02:00:31.447063 14709 webserver.cc:533] Webserver started at http://127.0.71.126:44633/ using document root <none> and password file <none>
I20260929 02:00:31.447745 14709 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:31.447875 14709 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:31.448206 14709 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:31.450583 14709 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "32251aed803045a48b9d5498eebfc19c"
format_stamp: "Formatted at 2026-09-29 02:00:31 on dist-test-slave-vz23"
I20260929 02:00:31.451319 14709 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "32251aed803045a48b9d5498eebfc19c"
format_stamp: "Formatted at 2026-09-29 02:00:31 on dist-test-slave-vz23"
I20260929 02:00:31.456398 14709 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.006s
I20260929 02:00:31.459685 14724 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:31.461023 14709 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:00:31.461202 14709 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "32251aed803045a48b9d5498eebfc19c"
format_stamp: "Formatted at 2026-09-29 02:00:31 on dist-test-slave-vz23"
I20260929 02:00:31.461387 14709 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:31.485097 14709 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:31.488746 14709 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:31.489042 14709 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:31.510931 14709 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:44141
I20260929 02:00:31.510967 14775 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:44141 every 8 connection(s)
I20260929 02:00:31.512425 14709 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 02:00:31.516584 14776 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:00:31.520092 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 14709
I20260929 02:00:31.520283 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 02:00:31.527102 14776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32251aed803045a48b9d5498eebfc19c: Bootstrap starting.
I20260929 02:00:31.530498 14776 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 32251aed803045a48b9d5498eebfc19c: Neither blocks nor log segments found. Creating new log.
I20260929 02:00:31.531527 14776 log.cc:826] T 00000000000000000000000000000000 P 32251aed803045a48b9d5498eebfc19c: Log is configured to *not* fsync() on all Append() calls
Sep 29 02:00:31 dist-test-slave-vz23 krb5kdc[14693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647230, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 02:00:31.534613 14776 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 32251aed803045a48b9d5498eebfc19c: No bootstrap required, opened a new log
I20260929 02:00:31.541379 14776 raft_consensus.cc:359] T 00000000000000000000000000000000 P 32251aed803045a48b9d5498eebfc19c [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32251aed803045a48b9d5498eebfc19c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44141 } }
I20260929 02:00:31.541740 14776 raft_consensus.cc:385] T 00000000000000000000000000000000 P 32251aed803045a48b9d5498eebfc19c [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:00:31.541893 14776 raft_consensus.cc:740] T 00000000000000000000000000000000 P 32251aed803045a48b9d5498eebfc19c [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 32251aed803045a48b9d5498eebfc19c, State: Initialized, Role: FOLLOWER
I20260929 02:00:31.542543 14776 consensus_queue.cc:260] T 00000000000000000000000000000000 P 32251aed803045a48b9d5498eebfc19c [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32251aed803045a48b9d5498eebfc19c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44141 } }
I20260929 02:00:31.542804 14776 raft_consensus.cc:399] T 00000000000000000000000000000000 P 32251aed803045a48b9d5498eebfc19c [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:00:31.542980 14776 raft_consensus.cc:493] T 00000000000000000000000000000000 P 32251aed803045a48b9d5498eebfc19c [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:00:31.543145 14776 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 32251aed803045a48b9d5498eebfc19c [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:00:31.545372 14776 raft_consensus.cc:515] T 00000000000000000000000000000000 P 32251aed803045a48b9d5498eebfc19c [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32251aed803045a48b9d5498eebfc19c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44141 } }
I20260929 02:00:31.545852 14776 leader_election.cc:304] T 00000000000000000000000000000000 P 32251aed803045a48b9d5498eebfc19c [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 32251aed803045a48b9d5498eebfc19c; no voters:
I20260929 02:00:31.546480 14776 leader_election.cc:290] T 00000000000000000000000000000000 P 32251aed803045a48b9d5498eebfc19c [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:00:31.546627 14781 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 32251aed803045a48b9d5498eebfc19c [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:00:31.547641 14781 raft_consensus.cc:697] T 00000000000000000000000000000000 P 32251aed803045a48b9d5498eebfc19c [term 1 LEADER]: Becoming Leader. State: Replica: 32251aed803045a48b9d5498eebfc19c, State: Running, Role: LEADER
I20260929 02:00:31.548135 14781 consensus_queue.cc:237] T 00000000000000000000000000000000 P 32251aed803045a48b9d5498eebfc19c [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32251aed803045a48b9d5498eebfc19c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44141 } }
I20260929 02:00:31.548707 14776 sys_catalog.cc:565] T 00000000000000000000000000000000 P 32251aed803045a48b9d5498eebfc19c [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:00:31.551191 14782 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32251aed803045a48b9d5498eebfc19c [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "32251aed803045a48b9d5498eebfc19c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32251aed803045a48b9d5498eebfc19c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44141 } } }
I20260929 02:00:31.551410 14783 sys_catalog.cc:455] T 00000000000000000000000000000000 P 32251aed803045a48b9d5498eebfc19c [sys.catalog]: SysCatalogTable state changed. Reason: New leader 32251aed803045a48b9d5498eebfc19c. Latest consensus state: current_term: 1 leader_uuid: "32251aed803045a48b9d5498eebfc19c" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "32251aed803045a48b9d5498eebfc19c" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44141 } } }
I20260929 02:00:31.552079 14782 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32251aed803045a48b9d5498eebfc19c [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:31.552138 14783 sys_catalog.cc:458] T 00000000000000000000000000000000 P 32251aed803045a48b9d5498eebfc19c [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:31.553906 14788 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 02:00:31.559226 14788 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 02:00:31.568161 14788 catalog_manager.cc:1396] Generated new cluster ID: 0bc39b853aa14b2d9e552e5c175ba5a6
I20260929 02:00:31.568297 14788 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 02:00:31.590440 14788 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 02:00:31.592125 14788 catalog_manager.cc:1553] Loading token signing keys...
I20260929 02:00:31.608537 14788 catalog_manager.cc:6219] T 00000000000000000000000000000000 P 32251aed803045a48b9d5498eebfc19c: Generated new TSK 0
I20260929 02:00:31.609406 14788 catalog_manager.cc:1563] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:31.748382 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:44141
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:42321
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:32.089784 14804 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:32.090122 14804 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:32.090302 14804 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:32.098827 14804 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:32.099071 14804 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 02:00:32.109972 14804 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:42321
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:44141
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:32.111622 14804 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:32.112957 14804 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:32.122934 14809 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:32.122967 14810 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:32.123839 14812 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:32.124583 14804 server_base.cc:1061] running on GCE node
I20260929 02:00:32.125149 14804 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:32.126098 14804 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:32.127302 14804 hybrid_clock.cc:648] HybridClock initialized: now 1790647232127244 us; error 64 us; skew 500 ppm
I20260929 02:00:32.127578 14804 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:32 dist-test-slave-vz23 krb5kdc[14693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:32.130823 14804 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 02:00:32.132354 14804 webserver.cc:533] Webserver started at http://127.0.71.65:33791/ using document root <none> and password file <none>
I20260929 02:00:32.133076 14804 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:32.133172 14804 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:32.133467 14804 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:32.135744 14804 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "41c80084a4644692ad7c477b462c3ae6"
format_stamp: "Formatted at 2026-09-29 02:00:32 on dist-test-slave-vz23"
I20260929 02:00:32.136533 14804 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "41c80084a4644692ad7c477b462c3ae6"
format_stamp: "Formatted at 2026-09-29 02:00:32 on dist-test-slave-vz23"
I20260929 02:00:32.141659 14804 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.002s sys 0.001s
I20260929 02:00:32.145118 14819 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:32.146555 14804 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:00:32.146698 14804 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "41c80084a4644692ad7c477b462c3ae6"
format_stamp: "Formatted at 2026-09-29 02:00:32 on dist-test-slave-vz23"
I20260929 02:00:32.146872 14804 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:32.161226 14804 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:32.165227 14804 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:32.165475 14804 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:32.166440 14804 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 02:00:32.167889 14804 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:00:32.167960 14804 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:32.168043 14804 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 02:00:32.168088 14804 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:32.202860 14804 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:41695
I20260929 02:00:32.202961 14931 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:41695 every 8 connection(s)
I20260929 02:00:32.204265 14804 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 02:00:32.209120 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 14804
I20260929 02:00:32.209317 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_4.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
Sep 29 02:00:32 dist-test-slave-vz23 krb5kdc[14693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647232, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 02:00:32.224722 14932 heartbeater.cc:344] Connected to a master server at 127.0.71.126:44141
I20260929 02:00:32.225090 14932 heartbeater.cc:461] Registering TS with master...
I20260929 02:00:32.225888 14932 heartbeater.cc:507] Master 127.0.71.126:44141 requested a full tablet report, sending...
I20260929 02:00:32.228027 14741 ts_manager.cc:194] Registered new tserver with Master: 41c80084a4644692ad7c477b462c3ae6 (127.0.71.65:41695)
I20260929 02:00:32.229635 14741 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:41923
I20260929 02:00:32.237414 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Sep 29 02:00:32 dist-test-slave-vz23 krb5kdc[14693](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:00:32 dist-test-slave-vz23 krb5kdc[14693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.0.71.126@KRBTEST.COM
I20260929 02:00:32.290952 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 14804
Sep 29 02:00:32 dist-test-slave-vz23 krb5kdc[14693](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647232, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 02:00:32.307018 14927 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:00:32.393776 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 14804
I20260929 02:00:32.410655 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 14709
I20260929 02:00:32.411813 14771 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:00:32.520855 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 14709
2026-09-29T02:00:32Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/4 (3873 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/5
Loading random data
Initializing database '/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:00:32 dist-test-slave-vz23 krb5kdc[14945](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:00:32 dist-test-slave-vz23 krb5kdc[14945](info): set up 2 sockets
Sep 29 02:00:32 dist-test-slave-vz23 krb5kdc[14945](info): commencing operation
krb5kdc: starting...
W20260929 02:00:34.617038 285 mini_kdc.cc:121] Time spent starting KDC: real 2.056s user 0.007s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 02:00:34 dist-test-slave-vz23 krb5kdc[14945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:00:34Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:00:34Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:34.945652 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:35341
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:36829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:35341
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:35.288533 14961 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:35.288902 14961 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:35.289072 14961 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:35.297537 14961 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:00:35.297647 14961 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:35.297777 14961 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:00:35.297873 14961 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:00:35.308674 14961 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:36829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:35341
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:35341
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:35.310247 14961 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:35.311542 14961 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:35.319389 14967 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:35.319416 14966 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:35.319988 14969 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:35.320744 14961 server_base.cc:1061] running on GCE node
I20260929 02:00:35.321499 14961 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:35.322852 14961 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:35.324092 14961 hybrid_clock.cc:648] HybridClock initialized: now 1790647235324063 us; error 40 us; skew 500 ppm
I20260929 02:00:35.324481 14961 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:35 dist-test-slave-vz23 krb5kdc[14945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647235, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:35.327459 14961 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 02:00:35.329126 14961 webserver.cc:533] Webserver started at http://127.0.71.126:35985/ using document root <none> and password file <none>
I20260929 02:00:35.329874 14961 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:35.330062 14961 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:35.330454 14961 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:35.332841 14961 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "f19b105645d04f05937e7f07bf24da61"
format_stamp: "Formatted at 2026-09-29 02:00:35 on dist-test-slave-vz23"
I20260929 02:00:35.333596 14961 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "f19b105645d04f05937e7f07bf24da61"
format_stamp: "Formatted at 2026-09-29 02:00:35 on dist-test-slave-vz23"
I20260929 02:00:35.338766 14961 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:00:35.342085 14976 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:35.343380 14961 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:00:35.343600 14961 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "f19b105645d04f05937e7f07bf24da61"
format_stamp: "Formatted at 2026-09-29 02:00:35 on dist-test-slave-vz23"
I20260929 02:00:35.343797 14961 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:35.417763 14961 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:35.430215 14961 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:35.430526 14961 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:35.451581 14961 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:35341
I20260929 02:00:35.451627 15027 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:35341 every 8 connection(s)
I20260929 02:00:35.453204 14961 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 02:00:35.457383 15028 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:00:35.462635 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 14961
I20260929 02:00:35.462831 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 02:00:35.467274 15028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f19b105645d04f05937e7f07bf24da61: Bootstrap starting.
I20260929 02:00:35.470432 15028 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P f19b105645d04f05937e7f07bf24da61: Neither blocks nor log segments found. Creating new log.
I20260929 02:00:35.471463 15028 log.cc:826] T 00000000000000000000000000000000 P f19b105645d04f05937e7f07bf24da61: Log is configured to *not* fsync() on all Append() calls
I20260929 02:00:35.474164 15028 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P f19b105645d04f05937e7f07bf24da61: No bootstrap required, opened a new log
Sep 29 02:00:35 dist-test-slave-vz23 krb5kdc[14945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647234, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 02:00:35.481601 15028 raft_consensus.cc:359] T 00000000000000000000000000000000 P f19b105645d04f05937e7f07bf24da61 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19b105645d04f05937e7f07bf24da61" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35341 } }
I20260929 02:00:35.482053 15028 raft_consensus.cc:385] T 00000000000000000000000000000000 P f19b105645d04f05937e7f07bf24da61 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:00:35.482184 15028 raft_consensus.cc:740] T 00000000000000000000000000000000 P f19b105645d04f05937e7f07bf24da61 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: f19b105645d04f05937e7f07bf24da61, State: Initialized, Role: FOLLOWER
I20260929 02:00:35.483002 15028 consensus_queue.cc:260] T 00000000000000000000000000000000 P f19b105645d04f05937e7f07bf24da61 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19b105645d04f05937e7f07bf24da61" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35341 } }
I20260929 02:00:35.483212 15028 raft_consensus.cc:399] T 00000000000000000000000000000000 P f19b105645d04f05937e7f07bf24da61 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:00:35.483382 15028 raft_consensus.cc:493] T 00000000000000000000000000000000 P f19b105645d04f05937e7f07bf24da61 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:00:35.483568 15028 raft_consensus.cc:3060] T 00000000000000000000000000000000 P f19b105645d04f05937e7f07bf24da61 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:00:35.486492 15028 raft_consensus.cc:515] T 00000000000000000000000000000000 P f19b105645d04f05937e7f07bf24da61 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19b105645d04f05937e7f07bf24da61" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35341 } }
I20260929 02:00:35.486963 15028 leader_election.cc:304] T 00000000000000000000000000000000 P f19b105645d04f05937e7f07bf24da61 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: f19b105645d04f05937e7f07bf24da61; no voters:
I20260929 02:00:35.487579 15028 leader_election.cc:290] T 00000000000000000000000000000000 P f19b105645d04f05937e7f07bf24da61 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:00:35.487737 15033 raft_consensus.cc:2804] T 00000000000000000000000000000000 P f19b105645d04f05937e7f07bf24da61 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:00:35.488464 15033 raft_consensus.cc:697] T 00000000000000000000000000000000 P f19b105645d04f05937e7f07bf24da61 [term 1 LEADER]: Becoming Leader. State: Replica: f19b105645d04f05937e7f07bf24da61, State: Running, Role: LEADER
I20260929 02:00:35.489002 15033 consensus_queue.cc:237] T 00000000000000000000000000000000 P f19b105645d04f05937e7f07bf24da61 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19b105645d04f05937e7f07bf24da61" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35341 } }
I20260929 02:00:35.489617 15028 sys_catalog.cc:565] T 00000000000000000000000000000000 P f19b105645d04f05937e7f07bf24da61 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:00:35.491977 15034 sys_catalog.cc:455] T 00000000000000000000000000000000 P f19b105645d04f05937e7f07bf24da61 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "f19b105645d04f05937e7f07bf24da61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19b105645d04f05937e7f07bf24da61" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35341 } } }
I20260929 02:00:35.492343 15034 sys_catalog.cc:458] T 00000000000000000000000000000000 P f19b105645d04f05937e7f07bf24da61 [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:35.493541 15035 sys_catalog.cc:455] T 00000000000000000000000000000000 P f19b105645d04f05937e7f07bf24da61 [sys.catalog]: SysCatalogTable state changed. Reason: New leader f19b105645d04f05937e7f07bf24da61. Latest consensus state: current_term: 1 leader_uuid: "f19b105645d04f05937e7f07bf24da61" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "f19b105645d04f05937e7f07bf24da61" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 35341 } } }
I20260929 02:00:35.493794 15035 sys_catalog.cc:458] T 00000000000000000000000000000000 P f19b105645d04f05937e7f07bf24da61 [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:35.494369 15039 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 02:00:35.499846 15039 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 02:00:35.509099 15039 catalog_manager.cc:1396] Generated new cluster ID: 20783019e3ff49b3874003c43f1d4d80
I20260929 02:00:35.509227 15039 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 02:00:35.544564 15039 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 02:00:35.545855 15039 catalog_manager.cc:1553] Loading token signing keys...
I20260929 02:00:35.556126 15039 catalog_manager.cc:6219] T 00000000000000000000000000000000 P f19b105645d04f05937e7f07bf24da61: Generated new TSK 0
I20260929 02:00:35.557005 15039 catalog_manager.cc:1563] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:35.682015 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:35341
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:36829
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:36.029614 15056 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:36.029935 15056 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:36.030042 15056 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:36.039415 15056 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:36.039609 15056 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 02:00:36.050794 15056 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:36829
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:35341
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:36.052343 15056 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:36.053665 15056 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:36.063560 15062 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:36.064056 15061 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:36.064380 15064 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:36.065491 15056 server_base.cc:1061] running on GCE node
I20260929 02:00:36.066133 15056 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:36.067121 15056 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:36.068360 15056 hybrid_clock.cc:648] HybridClock initialized: now 1790647236068324 us; error 50 us; skew 500 ppm
I20260929 02:00:36.068728 15056 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:36 dist-test-slave-vz23 krb5kdc[14945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:36.071879 15056 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 02:00:36.073542 15056 webserver.cc:533] Webserver started at http://127.0.71.65:42833/ using document root <none> and password file <none>
I20260929 02:00:36.074234 15056 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:36.074324 15056 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:36.074625 15056 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:36.077039 15056 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "92e94d77d7444782a80a57a4a0a5e4b3"
format_stamp: "Formatted at 2026-09-29 02:00:36 on dist-test-slave-vz23"
I20260929 02:00:36.077759 15056 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "92e94d77d7444782a80a57a4a0a5e4b3"
format_stamp: "Formatted at 2026-09-29 02:00:36 on dist-test-slave-vz23"
I20260929 02:00:36.083086 15056 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.000s sys 0.005s
I20260929 02:00:36.086364 15071 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:36.087657 15056 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.000s
I20260929 02:00:36.087831 15056 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "92e94d77d7444782a80a57a4a0a5e4b3"
format_stamp: "Formatted at 2026-09-29 02:00:36 on dist-test-slave-vz23"
I20260929 02:00:36.088024 15056 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:36.107447 15056 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:36.111532 15056 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:36.111867 15056 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:36.112829 15056 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 02:00:36.114342 15056 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:00:36.114415 15056 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:36.114583 15056 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 02:00:36.114656 15056 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:36.149420 15056 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:38635
I20260929 02:00:36.149468 15183 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:38635 every 8 connection(s)
I20260929 02:00:36.151080 15056 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 02:00:36.158681 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 15056
I20260929 02:00:36.158861 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_5.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
Sep 29 02:00:36 dist-test-slave-vz23 krb5kdc[14945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647236, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 02:00:36.170152 15184 heartbeater.cc:344] Connected to a master server at 127.0.71.126:35341
I20260929 02:00:36.170457 15184 heartbeater.cc:461] Registering TS with master...
I20260929 02:00:36.171245 15184 heartbeater.cc:507] Master 127.0.71.126:35341 requested a full tablet report, sending...
I20260929 02:00:36.173338 14993 ts_manager.cc:194] Registered new tserver with Master: 92e94d77d7444782a80a57a4a0a5e4b3 (127.0.71.65:38635)
I20260929 02:00:36.174523 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 02:00:36.175114 14993 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:33583
Sep 29 02:00:36 dist-test-slave-vz23 krb5kdc[14945](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:00:36 dist-test-slave-vz23 krb5kdc[14945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.0.71.126@KRBTEST.COM
I20260929 02:00:36.226779 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 15056
Sep 29 02:00:36 dist-test-slave-vz23 krb5kdc[14945](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647236, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 02:00:36.242509 15179 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:00:36.330075 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 15056
I20260929 02:00:36.347268 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 14961
I20260929 02:00:36.348364 15023 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:00:36.458812 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 14961
2026-09-29T02:00:36Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/5 (3938 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/6
Loading random data
Initializing database '/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:00:36 dist-test-slave-vz23 krb5kdc[15197](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:00:36 dist-test-slave-vz23 krb5kdc[15197](info): set up 2 sockets
Sep 29 02:00:36 dist-test-slave-vz23 krb5kdc[15197](info): commencing operation
krb5kdc: starting...
W20260929 02:00:38.597396 285 mini_kdc.cc:121] Time spent starting KDC: real 2.097s user 0.005s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 02:00:38 dist-test-slave-vz23 krb5kdc[15197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:00:38Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:00:38Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:38.925346 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:38673
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:40509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:38673
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:39.295064 15213 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:39.295441 15213 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:39.295595 15213 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:39.304134 15213 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:00:39.304301 15213 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:39.304427 15213 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:00:39.304538 15213 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:00:39.315477 15213 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:40509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:38673
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:38673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:39.317121 15213 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:39.318470 15213 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:39.326617 15218 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:39.326661 15219 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:39.327570 15221 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:39.327968 15213 server_base.cc:1061] running on GCE node
I20260929 02:00:39.328817 15213 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:39.330173 15213 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:39.331455 15213 hybrid_clock.cc:648] HybridClock initialized: now 1790647239331411 us; error 51 us; skew 500 ppm
I20260929 02:00:39.331902 15213 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:39 dist-test-slave-vz23 krb5kdc[15197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647239, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:39.335675 15213 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 02:00:39.337615 15213 webserver.cc:533] Webserver started at http://127.0.71.126:34921/ using document root <none> and password file <none>
I20260929 02:00:39.338344 15213 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:39.338482 15213 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:39.338840 15213 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:39.341339 15213 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "75c2ae2c2b174a758c64fdeaa721df1b"
format_stamp: "Formatted at 2026-09-29 02:00:39 on dist-test-slave-vz23"
I20260929 02:00:39.342080 15213 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "75c2ae2c2b174a758c64fdeaa721df1b"
format_stamp: "Formatted at 2026-09-29 02:00:39 on dist-test-slave-vz23"
I20260929 02:00:39.347385 15213 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:00:39.350790 15228 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:39.352180 15213 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 02:00:39.352330 15213 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "75c2ae2c2b174a758c64fdeaa721df1b"
format_stamp: "Formatted at 2026-09-29 02:00:39 on dist-test-slave-vz23"
I20260929 02:00:39.352563 15213 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:39.368822 15213 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:39.372768 15213 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:39.373003 15213 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:39.393908 15213 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:38673
I20260929 02:00:39.393939 15279 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:38673 every 8 connection(s)
I20260929 02:00:39.395464 15213 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 02:00:39.399763 15280 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:00:39.402463 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 15213
I20260929 02:00:39.402683 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 02:00:39.411609 15280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75c2ae2c2b174a758c64fdeaa721df1b: Bootstrap starting.
I20260929 02:00:39.415285 15280 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 75c2ae2c2b174a758c64fdeaa721df1b: Neither blocks nor log segments found. Creating new log.
Sep 29 02:00:39 dist-test-slave-vz23 krb5kdc[15197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647238, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 02:00:39.416548 15280 log.cc:826] T 00000000000000000000000000000000 P 75c2ae2c2b174a758c64fdeaa721df1b: Log is configured to *not* fsync() on all Append() calls
I20260929 02:00:39.419517 15280 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 75c2ae2c2b174a758c64fdeaa721df1b: No bootstrap required, opened a new log
I20260929 02:00:39.426198 15280 raft_consensus.cc:359] T 00000000000000000000000000000000 P 75c2ae2c2b174a758c64fdeaa721df1b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c2ae2c2b174a758c64fdeaa721df1b" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38673 } }
I20260929 02:00:39.426617 15280 raft_consensus.cc:385] T 00000000000000000000000000000000 P 75c2ae2c2b174a758c64fdeaa721df1b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:00:39.426741 15280 raft_consensus.cc:740] T 00000000000000000000000000000000 P 75c2ae2c2b174a758c64fdeaa721df1b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 75c2ae2c2b174a758c64fdeaa721df1b, State: Initialized, Role: FOLLOWER
I20260929 02:00:39.427397 15280 consensus_queue.cc:260] T 00000000000000000000000000000000 P 75c2ae2c2b174a758c64fdeaa721df1b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c2ae2c2b174a758c64fdeaa721df1b" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38673 } }
I20260929 02:00:39.427649 15280 raft_consensus.cc:399] T 00000000000000000000000000000000 P 75c2ae2c2b174a758c64fdeaa721df1b [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:00:39.427839 15280 raft_consensus.cc:493] T 00000000000000000000000000000000 P 75c2ae2c2b174a758c64fdeaa721df1b [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:00:39.428063 15280 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 75c2ae2c2b174a758c64fdeaa721df1b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:00:39.430624 15280 raft_consensus.cc:515] T 00000000000000000000000000000000 P 75c2ae2c2b174a758c64fdeaa721df1b [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c2ae2c2b174a758c64fdeaa721df1b" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38673 } }
I20260929 02:00:39.431182 15280 leader_election.cc:304] T 00000000000000000000000000000000 P 75c2ae2c2b174a758c64fdeaa721df1b [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 75c2ae2c2b174a758c64fdeaa721df1b; no voters:
I20260929 02:00:39.431819 15280 leader_election.cc:290] T 00000000000000000000000000000000 P 75c2ae2c2b174a758c64fdeaa721df1b [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:00:39.431952 15285 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 75c2ae2c2b174a758c64fdeaa721df1b [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:00:39.433364 15285 raft_consensus.cc:697] T 00000000000000000000000000000000 P 75c2ae2c2b174a758c64fdeaa721df1b [term 1 LEADER]: Becoming Leader. State: Replica: 75c2ae2c2b174a758c64fdeaa721df1b, State: Running, Role: LEADER
I20260929 02:00:39.433959 15285 consensus_queue.cc:237] T 00000000000000000000000000000000 P 75c2ae2c2b174a758c64fdeaa721df1b [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c2ae2c2b174a758c64fdeaa721df1b" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38673 } }
I20260929 02:00:39.434412 15280 sys_catalog.cc:565] T 00000000000000000000000000000000 P 75c2ae2c2b174a758c64fdeaa721df1b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:00:39.436890 15286 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75c2ae2c2b174a758c64fdeaa721df1b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "75c2ae2c2b174a758c64fdeaa721df1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c2ae2c2b174a758c64fdeaa721df1b" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38673 } } }
I20260929 02:00:39.437222 15286 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75c2ae2c2b174a758c64fdeaa721df1b [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:39.437139 15287 sys_catalog.cc:455] T 00000000000000000000000000000000 P 75c2ae2c2b174a758c64fdeaa721df1b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 75c2ae2c2b174a758c64fdeaa721df1b. Latest consensus state: current_term: 1 leader_uuid: "75c2ae2c2b174a758c64fdeaa721df1b" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "75c2ae2c2b174a758c64fdeaa721df1b" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38673 } } }
I20260929 02:00:39.437404 15287 sys_catalog.cc:458] T 00000000000000000000000000000000 P 75c2ae2c2b174a758c64fdeaa721df1b [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:39.439566 15293 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 02:00:39.445154 15293 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 02:00:39.455339 15293 catalog_manager.cc:1396] Generated new cluster ID: 671fbb3930e04e3792735c409c6aee5d
I20260929 02:00:39.455470 15293 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 02:00:39.480837 15293 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 02:00:39.482599 15293 catalog_manager.cc:1553] Loading token signing keys...
I20260929 02:00:39.492030 15293 catalog_manager.cc:6219] T 00000000000000000000000000000000 P 75c2ae2c2b174a758c64fdeaa721df1b: Generated new TSK 0
I20260929 02:00:39.493072 15293 catalog_manager.cc:1563] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:39.615301 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:38673
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:40509
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:39.963014 15308 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:39.963371 15308 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:39.963555 15308 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:39.972342 15308 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:39.972666 15308 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 02:00:39.983664 15308 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:40509
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:38673
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:39.985387 15308 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:39.986873 15308 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:39.996915 15313 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:39.997853 15314 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:39.998986 15316 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:39.999140 15308 server_base.cc:1061] running on GCE node
I20260929 02:00:39.999892 15308 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:40.000926 15308 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:40.002147 15308 hybrid_clock.cc:648] HybridClock initialized: now 1790647240002115 us; error 33 us; skew 500 ppm
I20260929 02:00:40.002516 15308 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:40 dist-test-slave-vz23 krb5kdc[15197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:40.005700 15308 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 02:00:40.007309 15308 webserver.cc:533] Webserver started at http://127.0.71.65:36403/ using document root <none> and password file <none>
I20260929 02:00:40.008028 15308 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:40.008163 15308 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:40.008585 15308 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:40.010948 15308 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "2c376559b3f54e3da32cbe480a50eeb0"
format_stamp: "Formatted at 2026-09-29 02:00:40 on dist-test-slave-vz23"
I20260929 02:00:40.011766 15308 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "2c376559b3f54e3da32cbe480a50eeb0"
format_stamp: "Formatted at 2026-09-29 02:00:40 on dist-test-slave-vz23"
I20260929 02:00:40.017071 15308 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.002s
I20260929 02:00:40.020287 15323 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:40.021624 15308 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:00:40.021798 15308 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "2c376559b3f54e3da32cbe480a50eeb0"
format_stamp: "Formatted at 2026-09-29 02:00:40 on dist-test-slave-vz23"
I20260929 02:00:40.021987 15308 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:40.047955 15308 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:40.052125 15308 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:40.052491 15308 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:40.053479 15308 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 02:00:40.055087 15308 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:00:40.055199 15308 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:40.055351 15308 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 02:00:40.055425 15308 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:40.090346 15308 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:39617
I20260929 02:00:40.090433 15435 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:39617 every 8 connection(s)
I20260929 02:00:40.091765 15308 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 02:00:40.092867 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 15308
I20260929 02:00:40.093057 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_6.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
Sep 29 02:00:40 dist-test-slave-vz23 krb5kdc[15197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647240, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 02:00:40.112906 15436 heartbeater.cc:344] Connected to a master server at 127.0.71.126:38673
I20260929 02:00:40.113281 15436 heartbeater.cc:461] Registering TS with master...
I20260929 02:00:40.114069 15436 heartbeater.cc:507] Master 127.0.71.126:38673 requested a full tablet report, sending...
I20260929 02:00:40.116355 15245 ts_manager.cc:194] Registered new tserver with Master: 2c376559b3f54e3da32cbe480a50eeb0 (127.0.71.65:39617)
I20260929 02:00:40.117964 15245 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:59783
I20260929 02:00:40.121658 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Sep 29 02:00:40 dist-test-slave-vz23 krb5kdc[15197](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:00:40 dist-test-slave-vz23 krb5kdc[15197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.0.71.126@KRBTEST.COM
I20260929 02:00:40.175587 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 15308
Sep 29 02:00:40 dist-test-slave-vz23 krb5kdc[15197](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647240, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 02:00:40.192536 15431 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:00:40.280980 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 15308
I20260929 02:00:40.298164 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 15213
I20260929 02:00:40.299475 15275 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:00:40.407088 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 15213
2026-09-29T02:00:40Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/6 (3947 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/7
Loading random data
Initializing database '/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:00:40 dist-test-slave-vz23 krb5kdc[15449](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:00:40 dist-test-slave-vz23 krb5kdc[15449](info): set up 2 sockets
Sep 29 02:00:40 dist-test-slave-vz23 krb5kdc[15449](info): commencing operation
krb5kdc: starting...
W20260929 02:00:42.565362 285 mini_kdc.cc:121] Time spent starting KDC: real 2.117s user 0.002s sys 0.025s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 02:00:42 dist-test-slave-vz23 krb5kdc[15449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:00:42Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:00:42Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:42.897843 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:38627
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:34157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:38627
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:43.243028 15465 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:43.243407 15465 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:43.243566 15465 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:43.252315 15465 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:00:43.252490 15465 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:43.252614 15465 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:00:43.252710 15465 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:00:43.263859 15465 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:34157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:38627
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:38627
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:43.265538 15465 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:43.266935 15465 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:43.275427 15471 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:43.275456 15470 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:43.276505 15473 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:43.277010 15465 server_base.cc:1061] running on GCE node
I20260929 02:00:43.277789 15465 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:43.279284 15465 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:43.280560 15465 hybrid_clock.cc:648] HybridClock initialized: now 1790647243280508 us; error 47 us; skew 500 ppm
I20260929 02:00:43.280921 15465 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:43 dist-test-slave-vz23 krb5kdc[15449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:43.284035 15465 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 02:00:43.285754 15465 webserver.cc:533] Webserver started at http://127.0.71.126:41497/ using document root <none> and password file <none>
I20260929 02:00:43.286492 15465 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:43.286631 15465 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:43.286989 15465 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:43.289525 15465 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "dcd806bc0c7940d48e2e6d9fe6aec148"
format_stamp: "Formatted at 2026-09-29 02:00:43 on dist-test-slave-vz23"
I20260929 02:00:43.290299 15465 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "dcd806bc0c7940d48e2e6d9fe6aec148"
format_stamp: "Formatted at 2026-09-29 02:00:43 on dist-test-slave-vz23"
I20260929 02:00:43.295703 15465 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:00:43.299118 15480 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:43.300405 15465 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:00:43.300673 15465 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "dcd806bc0c7940d48e2e6d9fe6aec148"
format_stamp: "Formatted at 2026-09-29 02:00:43 on dist-test-slave-vz23"
I20260929 02:00:43.300905 15465 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:43.323047 15465 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:43.327059 15465 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:43.327363 15465 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:43.348308 15465 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:38627
I20260929 02:00:43.348351 15531 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:38627 every 8 connection(s)
I20260929 02:00:43.349874 15465 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 02:00:43.354254 15532 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:00:43.354748 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 15465
I20260929 02:00:43.354933 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 02:00:43.367678 15532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcd806bc0c7940d48e2e6d9fe6aec148: Bootstrap starting.
Sep 29 02:00:43 dist-test-slave-vz23 krb5kdc[15449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647242, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 02:00:43.371196 15532 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P dcd806bc0c7940d48e2e6d9fe6aec148: Neither blocks nor log segments found. Creating new log.
I20260929 02:00:43.372310 15532 log.cc:826] T 00000000000000000000000000000000 P dcd806bc0c7940d48e2e6d9fe6aec148: Log is configured to *not* fsync() on all Append() calls
I20260929 02:00:43.375056 15532 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P dcd806bc0c7940d48e2e6d9fe6aec148: No bootstrap required, opened a new log
I20260929 02:00:43.380841 15532 raft_consensus.cc:359] T 00000000000000000000000000000000 P dcd806bc0c7940d48e2e6d9fe6aec148 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcd806bc0c7940d48e2e6d9fe6aec148" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38627 } }
I20260929 02:00:43.381207 15532 raft_consensus.cc:385] T 00000000000000000000000000000000 P dcd806bc0c7940d48e2e6d9fe6aec148 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:00:43.381348 15532 raft_consensus.cc:740] T 00000000000000000000000000000000 P dcd806bc0c7940d48e2e6d9fe6aec148 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: dcd806bc0c7940d48e2e6d9fe6aec148, State: Initialized, Role: FOLLOWER
I20260929 02:00:43.382040 15532 consensus_queue.cc:260] T 00000000000000000000000000000000 P dcd806bc0c7940d48e2e6d9fe6aec148 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcd806bc0c7940d48e2e6d9fe6aec148" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38627 } }
I20260929 02:00:43.382270 15532 raft_consensus.cc:399] T 00000000000000000000000000000000 P dcd806bc0c7940d48e2e6d9fe6aec148 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:00:43.382405 15532 raft_consensus.cc:493] T 00000000000000000000000000000000 P dcd806bc0c7940d48e2e6d9fe6aec148 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:00:43.382589 15532 raft_consensus.cc:3060] T 00000000000000000000000000000000 P dcd806bc0c7940d48e2e6d9fe6aec148 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:00:43.384968 15532 raft_consensus.cc:515] T 00000000000000000000000000000000 P dcd806bc0c7940d48e2e6d9fe6aec148 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcd806bc0c7940d48e2e6d9fe6aec148" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38627 } }
I20260929 02:00:43.385506 15532 leader_election.cc:304] T 00000000000000000000000000000000 P dcd806bc0c7940d48e2e6d9fe6aec148 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: dcd806bc0c7940d48e2e6d9fe6aec148; no voters:
I20260929 02:00:43.386224 15532 leader_election.cc:290] T 00000000000000000000000000000000 P dcd806bc0c7940d48e2e6d9fe6aec148 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:00:43.386327 15537 raft_consensus.cc:2804] T 00000000000000000000000000000000 P dcd806bc0c7940d48e2e6d9fe6aec148 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:00:43.387080 15537 raft_consensus.cc:697] T 00000000000000000000000000000000 P dcd806bc0c7940d48e2e6d9fe6aec148 [term 1 LEADER]: Becoming Leader. State: Replica: dcd806bc0c7940d48e2e6d9fe6aec148, State: Running, Role: LEADER
I20260929 02:00:43.387619 15537 consensus_queue.cc:237] T 00000000000000000000000000000000 P dcd806bc0c7940d48e2e6d9fe6aec148 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcd806bc0c7940d48e2e6d9fe6aec148" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38627 } }
I20260929 02:00:43.388593 15532 sys_catalog.cc:565] T 00000000000000000000000000000000 P dcd806bc0c7940d48e2e6d9fe6aec148 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:00:43.390926 15539 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcd806bc0c7940d48e2e6d9fe6aec148 [sys.catalog]: SysCatalogTable state changed. Reason: New leader dcd806bc0c7940d48e2e6d9fe6aec148. Latest consensus state: current_term: 1 leader_uuid: "dcd806bc0c7940d48e2e6d9fe6aec148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcd806bc0c7940d48e2e6d9fe6aec148" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38627 } } }
I20260929 02:00:43.391433 15539 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcd806bc0c7940d48e2e6d9fe6aec148 [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:43.391117 15538 sys_catalog.cc:455] T 00000000000000000000000000000000 P dcd806bc0c7940d48e2e6d9fe6aec148 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "dcd806bc0c7940d48e2e6d9fe6aec148" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "dcd806bc0c7940d48e2e6d9fe6aec148" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 38627 } } }
I20260929 02:00:43.394239 15544 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 02:00:43.394352 15538 sys_catalog.cc:458] T 00000000000000000000000000000000 P dcd806bc0c7940d48e2e6d9fe6aec148 [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:43.400282 15544 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 02:00:43.409613 15544 catalog_manager.cc:1396] Generated new cluster ID: 55af221ca6a64329a27cd1012dfdf7fb
I20260929 02:00:43.409749 15544 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 02:00:43.426785 15544 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 02:00:43.428480 15544 catalog_manager.cc:1553] Loading token signing keys...
I20260929 02:00:43.444365 15544 catalog_manager.cc:6219] T 00000000000000000000000000000000 P dcd806bc0c7940d48e2e6d9fe6aec148: Generated new TSK 0
I20260929 02:00:43.445540 15544 catalog_manager.cc:1563] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:43.589464 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:38627
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:34157
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:43.934297 15560 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:43.934689 15560 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:43.934789 15560 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:43.943396 15560 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:43.943706 15560 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 02:00:43.955075 15560 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:34157
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:38627
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:43.956672 15560 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:43.957957 15560 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:43.967299 15566 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:43.967844 15568 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:43.967324 15565 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:43.968767 15560 server_base.cc:1061] running on GCE node
I20260929 02:00:43.969336 15560 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:43.970216 15560 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:43.971470 15560 hybrid_clock.cc:648] HybridClock initialized: now 1790647243971420 us; error 57 us; skew 500 ppm
I20260929 02:00:43.971835 15560 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:43 dist-test-slave-vz23 krb5kdc[15449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:43.975104 15560 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 02:00:43.976801 15560 webserver.cc:533] Webserver started at http://127.0.71.65:45039/ using document root <none> and password file <none>
I20260929 02:00:43.977519 15560 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:43.977653 15560 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:43.977998 15560 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:43.980357 15560 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "d1e8fa748fc544849f97a0447b717278"
format_stamp: "Formatted at 2026-09-29 02:00:43 on dist-test-slave-vz23"
I20260929 02:00:43.981179 15560 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "d1e8fa748fc544849f97a0447b717278"
format_stamp: "Formatted at 2026-09-29 02:00:43 on dist-test-slave-vz23"
I20260929 02:00:43.986485 15560 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.000s
I20260929 02:00:43.989866 15575 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:43.991323 15560 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:00:43.991542 15560 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "d1e8fa748fc544849f97a0447b717278"
format_stamp: "Formatted at 2026-09-29 02:00:43 on dist-test-slave-vz23"
I20260929 02:00:43.991778 15560 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:44.014513 15560 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:44.018637 15560 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:44.018961 15560 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:44.020105 15560 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 02:00:44.021860 15560 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:00:44.021976 15560 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:44.022150 15560 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 02:00:44.022234 15560 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:44.060284 15560 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:39001
I20260929 02:00:44.060379 15687 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:39001 every 8 connection(s)
I20260929 02:00:44.061812 15560 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 02:00:44.067369 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 15560
I20260929 02:00:44.067526 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_7.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
Sep 29 02:00:44 dist-test-slave-vz23 krb5kdc[15449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647243, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 02:00:44.084226 15688 heartbeater.cc:344] Connected to a master server at 127.0.71.126:38627
I20260929 02:00:44.084648 15688 heartbeater.cc:461] Registering TS with master...
I20260929 02:00:44.085490 15688 heartbeater.cc:507] Master 127.0.71.126:38627 requested a full tablet report, sending...
I20260929 02:00:44.087706 15497 ts_manager.cc:194] Registered new tserver with Master: d1e8fa748fc544849f97a0447b717278 (127.0.71.65:39001)
I20260929 02:00:44.089315 15497 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:43531
I20260929 02:00:44.095696 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Sep 29 02:00:44 dist-test-slave-vz23 krb5kdc[15449](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:00:44 dist-test-slave-vz23 krb5kdc[15449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.0.71.126@KRBTEST.COM
I20260929 02:00:44.150907 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 15560
Sep 29 02:00:44 dist-test-slave-vz23 krb5kdc[15449](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647244, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 02:00:44.167415 15683 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:00:44.254717 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 15560
I20260929 02:00:44.272044 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 15465
I20260929 02:00:44.273375 15527 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:00:44.380736 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 15465
2026-09-29T02:00:44Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/7 (3974 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/8
Loading random data
Initializing database '/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:00:44 dist-test-slave-vz23 krb5kdc[15701](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:00:44 dist-test-slave-vz23 krb5kdc[15701](info): set up 2 sockets
Sep 29 02:00:44 dist-test-slave-vz23 krb5kdc[15701](info): commencing operation
krb5kdc: starting...
W20260929 02:00:46.581034 285 mini_kdc.cc:121] Time spent starting KDC: real 2.158s user 0.000s sys 0.029s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 02:00:46 dist-test-slave-vz23 krb5kdc[15701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:00:46Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:00:46Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:46.904577 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:36095
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:39571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:36095
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:47.261238 15717 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:47.261662 15717 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:47.261816 15717 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:47.271176 15717 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:00:47.271343 15717 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:47.271436 15717 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:00:47.271531 15717 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:00:47.283034 15717 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:39571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:36095
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:36095
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:47.284773 15717 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:47.286154 15717 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:47.294620 15722 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:47.295324 15723 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:47.295984 15717 server_base.cc:1061] running on GCE node
W20260929 02:00:47.296334 15725 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:47.297331 15717 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:47.298797 15717 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:47.300057 15717 hybrid_clock.cc:648] HybridClock initialized: now 1790647247299990 us; error 79 us; skew 500 ppm
I20260929 02:00:47.300487 15717 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:47 dist-test-slave-vz23 krb5kdc[15701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:47.303809 15717 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 02:00:47.305588 15717 webserver.cc:533] Webserver started at http://127.0.71.126:44583/ using document root <none> and password file <none>
I20260929 02:00:47.306358 15717 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:47.306509 15717 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:47.306861 15717 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:47.309466 15717 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "009a8d88ec8143aebdb852db54754aa2"
format_stamp: "Formatted at 2026-09-29 02:00:47 on dist-test-slave-vz23"
I20260929 02:00:47.310317 15717 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "009a8d88ec8143aebdb852db54754aa2"
format_stamp: "Formatted at 2026-09-29 02:00:47 on dist-test-slave-vz23"
I20260929 02:00:47.315930 15717 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:00:47.319592 15732 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:47.321040 15717 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.002s
I20260929 02:00:47.321245 15717 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "009a8d88ec8143aebdb852db54754aa2"
format_stamp: "Formatted at 2026-09-29 02:00:47 on dist-test-slave-vz23"
I20260929 02:00:47.321442 15717 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:47.336127 15717 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:47.345928 15717 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:47.346262 15717 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:47.367628 15717 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:36095
I20260929 02:00:47.367663 15783 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:36095 every 8 connection(s)
I20260929 02:00:47.369313 15717 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 02:00:47.371058 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 15717
I20260929 02:00:47.371268 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 02:00:47.374315 15784 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:00:47 dist-test-slave-vz23 krb5kdc[15701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647246, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 02:00:47.386480 15784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 009a8d88ec8143aebdb852db54754aa2: Bootstrap starting.
I20260929 02:00:47.390861 15784 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 009a8d88ec8143aebdb852db54754aa2: Neither blocks nor log segments found. Creating new log.
I20260929 02:00:47.392122 15784 log.cc:826] T 00000000000000000000000000000000 P 009a8d88ec8143aebdb852db54754aa2: Log is configured to *not* fsync() on all Append() calls
I20260929 02:00:47.395330 15784 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 009a8d88ec8143aebdb852db54754aa2: No bootstrap required, opened a new log
I20260929 02:00:47.401348 15784 raft_consensus.cc:359] T 00000000000000000000000000000000 P 009a8d88ec8143aebdb852db54754aa2 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009a8d88ec8143aebdb852db54754aa2" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36095 } }
I20260929 02:00:47.401734 15784 raft_consensus.cc:385] T 00000000000000000000000000000000 P 009a8d88ec8143aebdb852db54754aa2 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:00:47.401870 15784 raft_consensus.cc:740] T 00000000000000000000000000000000 P 009a8d88ec8143aebdb852db54754aa2 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 009a8d88ec8143aebdb852db54754aa2, State: Initialized, Role: FOLLOWER
I20260929 02:00:47.402460 15784 consensus_queue.cc:260] T 00000000000000000000000000000000 P 009a8d88ec8143aebdb852db54754aa2 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009a8d88ec8143aebdb852db54754aa2" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36095 } }
I20260929 02:00:47.402695 15784 raft_consensus.cc:399] T 00000000000000000000000000000000 P 009a8d88ec8143aebdb852db54754aa2 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:00:47.402840 15784 raft_consensus.cc:493] T 00000000000000000000000000000000 P 009a8d88ec8143aebdb852db54754aa2 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:00:47.403008 15784 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 009a8d88ec8143aebdb852db54754aa2 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:00:47.405630 15784 raft_consensus.cc:515] T 00000000000000000000000000000000 P 009a8d88ec8143aebdb852db54754aa2 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009a8d88ec8143aebdb852db54754aa2" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36095 } }
I20260929 02:00:47.406134 15784 leader_election.cc:304] T 00000000000000000000000000000000 P 009a8d88ec8143aebdb852db54754aa2 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 009a8d88ec8143aebdb852db54754aa2; no voters:
I20260929 02:00:47.406774 15784 leader_election.cc:290] T 00000000000000000000000000000000 P 009a8d88ec8143aebdb852db54754aa2 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:00:47.406911 15789 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 009a8d88ec8143aebdb852db54754aa2 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:00:47.407648 15789 raft_consensus.cc:697] T 00000000000000000000000000000000 P 009a8d88ec8143aebdb852db54754aa2 [term 1 LEADER]: Becoming Leader. State: Replica: 009a8d88ec8143aebdb852db54754aa2, State: Running, Role: LEADER
I20260929 02:00:47.408268 15789 consensus_queue.cc:237] T 00000000000000000000000000000000 P 009a8d88ec8143aebdb852db54754aa2 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009a8d88ec8143aebdb852db54754aa2" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36095 } }
I20260929 02:00:47.409099 15784 sys_catalog.cc:565] T 00000000000000000000000000000000 P 009a8d88ec8143aebdb852db54754aa2 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:00:47.411180 15791 sys_catalog.cc:455] T 00000000000000000000000000000000 P 009a8d88ec8143aebdb852db54754aa2 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 009a8d88ec8143aebdb852db54754aa2. Latest consensus state: current_term: 1 leader_uuid: "009a8d88ec8143aebdb852db54754aa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009a8d88ec8143aebdb852db54754aa2" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36095 } } }
I20260929 02:00:47.411315 15790 sys_catalog.cc:455] T 00000000000000000000000000000000 P 009a8d88ec8143aebdb852db54754aa2 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "009a8d88ec8143aebdb852db54754aa2" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "009a8d88ec8143aebdb852db54754aa2" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 36095 } } }
I20260929 02:00:47.411588 15790 sys_catalog.cc:458] T 00000000000000000000000000000000 P 009a8d88ec8143aebdb852db54754aa2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:47.411592 15791 sys_catalog.cc:458] T 00000000000000000000000000000000 P 009a8d88ec8143aebdb852db54754aa2 [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:47.413264 15794 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 02:00:47.419593 15794 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 02:00:47.428900 15794 catalog_manager.cc:1396] Generated new cluster ID: cb4bd593cf3a4459bbb068dfe5dc05a6
I20260929 02:00:47.429040 15794 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 02:00:47.449499 15794 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 02:00:47.451269 15794 catalog_manager.cc:1553] Loading token signing keys...
I20260929 02:00:47.474260 15794 catalog_manager.cc:6219] T 00000000000000000000000000000000 P 009a8d88ec8143aebdb852db54754aa2: Generated new TSK 0
I20260929 02:00:47.475404 15794 catalog_manager.cc:1563] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:47.605326 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:36095
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:39571
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:47.957644 15812 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:47.958002 15812 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:47.958163 15812 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:47.966446 15812 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:47.966696 15812 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 02:00:47.977641 15812 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:39571
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:36095
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:47.979195 15812 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:47.980492 15812 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:47.989847 15817 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:47.990101 15820 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:47.990167 15818 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:47.991470 15812 server_base.cc:1061] running on GCE node
I20260929 02:00:47.992075 15812 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:47.993053 15812 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:47.994287 15812 hybrid_clock.cc:648] HybridClock initialized: now 1790647247994239 us; error 43 us; skew 500 ppm
I20260929 02:00:47.994647 15812 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:47 dist-test-slave-vz23 krb5kdc[15701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:47.997771 15812 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 02:00:47.999364 15812 webserver.cc:533] Webserver started at http://127.0.71.65:33007/ using document root <none> and password file <none>
I20260929 02:00:48.000092 15812 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:48.000227 15812 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:48.000648 15812 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:48.003104 15812 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "b9de91b4bf2c4a99b0a45326799d8e5e"
format_stamp: "Formatted at 2026-09-29 02:00:47 on dist-test-slave-vz23"
I20260929 02:00:48.003866 15812 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "b9de91b4bf2c4a99b0a45326799d8e5e"
format_stamp: "Formatted at 2026-09-29 02:00:47 on dist-test-slave-vz23"
I20260929 02:00:48.009430 15812 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.008s sys 0.000s
I20260929 02:00:48.012959 15827 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:48.014303 15812 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:00:48.014477 15812 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "b9de91b4bf2c4a99b0a45326799d8e5e"
format_stamp: "Formatted at 2026-09-29 02:00:47 on dist-test-slave-vz23"
I20260929 02:00:48.014686 15812 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:48.027593 15812 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:48.031432 15812 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:48.031741 15812 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:48.032748 15812 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 02:00:48.034250 15812 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:00:48.034353 15812 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:48.034505 15812 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 02:00:48.034578 15812 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:48.069815 15812 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:43495
I20260929 02:00:48.069926 15939 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:43495 every 8 connection(s)
I20260929 02:00:48.071259 15812 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 02:00:48.075978 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 15812
I20260929 02:00:48.076169 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_8.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
Sep 29 02:00:48 dist-test-slave-vz23 krb5kdc[15701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647247, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 02:00:48.090848 15940 heartbeater.cc:344] Connected to a master server at 127.0.71.126:36095
I20260929 02:00:48.091308 15940 heartbeater.cc:461] Registering TS with master...
I20260929 02:00:48.092136 15940 heartbeater.cc:507] Master 127.0.71.126:36095 requested a full tablet report, sending...
I20260929 02:00:48.094296 15748 ts_manager.cc:194] Registered new tserver with Master: b9de91b4bf2c4a99b0a45326799d8e5e (127.0.71.65:43495)
I20260929 02:00:48.095851 15748 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:46579
I20260929 02:00:48.103677 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Sep 29 02:00:48 dist-test-slave-vz23 krb5kdc[15701](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:00:48 dist-test-slave-vz23 krb5kdc[15701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.0.71.126@KRBTEST.COM
I20260929 02:00:48.158463 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 15812
Sep 29 02:00:48 dist-test-slave-vz23 krb5kdc[15701](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647248, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 02:00:48.175354 15935 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:00:48.264037 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 15812
I20260929 02:00:48.281586 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 15717
I20260929 02:00:48.282675 15779 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:00:48.392330 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 15717
2026-09-29T02:00:48Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/8 (4008 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/9
Loading random data
Initializing database '/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:00:48 dist-test-slave-vz23 krb5kdc[15953](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:00:48 dist-test-slave-vz23 krb5kdc[15953](info): set up 2 sockets
Sep 29 02:00:48 dist-test-slave-vz23 krb5kdc[15953](info): commencing operation
krb5kdc: starting...
W20260929 02:00:50.489307 285 mini_kdc.cc:121] Time spent starting KDC: real 2.059s user 0.006s sys 0.022s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 02:00:50 dist-test-slave-vz23 krb5kdc[15953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:00:50Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:00:50Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:50.816506 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:44673
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:46549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:44673
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:51.178580 15969 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:51.178977 15969 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:51.179162 15969 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:51.188081 15969 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:00:51.188208 15969 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:51.188354 15969 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:00:51.188470 15969 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:00:51.199658 15969 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:46549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:44673
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:44673
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:51.201323 15969 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:51.202720 15969 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:51.211557 15975 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:51.212154 15977 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:51.212244 15974 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:51.213361 15969 server_base.cc:1061] running on GCE node
I20260929 02:00:51.214131 15969 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:51.215567 15969 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:51.216763 15969 hybrid_clock.cc:648] HybridClock initialized: now 1790647251216730 us; error 36 us; skew 500 ppm
I20260929 02:00:51.217110 15969 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:51 dist-test-slave-vz23 krb5kdc[15953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:51.220187 15969 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 02:00:51.221907 15969 webserver.cc:533] Webserver started at http://127.0.71.126:35279/ using document root <none> and password file <none>
I20260929 02:00:51.222627 15969 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:51.222764 15969 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:51.223104 15969 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:51.225464 15969 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "7486def25e8c4e67b74e71c6e10ae01f"
format_stamp: "Formatted at 2026-09-29 02:00:51 on dist-test-slave-vz23"
I20260929 02:00:51.226222 15969 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "7486def25e8c4e67b74e71c6e10ae01f"
format_stamp: "Formatted at 2026-09-29 02:00:51 on dist-test-slave-vz23"
I20260929 02:00:51.231463 15969 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.001s sys 0.005s
I20260929 02:00:51.234827 15984 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:51.236131 15969 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.002s sys 0.000s
I20260929 02:00:51.236307 15969 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "7486def25e8c4e67b74e71c6e10ae01f"
format_stamp: "Formatted at 2026-09-29 02:00:51 on dist-test-slave-vz23"
I20260929 02:00:51.236521 15969 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:51.265765 15969 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:51.270123 15969 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:51.270464 15969 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:51.293613 15969 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:44673
I20260929 02:00:51.293658 16035 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:44673 every 8 connection(s)
I20260929 02:00:51.295271 15969 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 02:00:51.295818 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 15969
I20260929 02:00:51.296051 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 02:00:51.300611 16036 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:00:51 dist-test-slave-vz23 krb5kdc[15953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647250, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 02:00:51.310364 16036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7486def25e8c4e67b74e71c6e10ae01f: Bootstrap starting.
I20260929 02:00:51.314357 16036 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7486def25e8c4e67b74e71c6e10ae01f: Neither blocks nor log segments found. Creating new log.
I20260929 02:00:51.315434 16036 log.cc:826] T 00000000000000000000000000000000 P 7486def25e8c4e67b74e71c6e10ae01f: Log is configured to *not* fsync() on all Append() calls
I20260929 02:00:51.318408 16036 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7486def25e8c4e67b74e71c6e10ae01f: No bootstrap required, opened a new log
I20260929 02:00:51.324119 16036 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7486def25e8c4e67b74e71c6e10ae01f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7486def25e8c4e67b74e71c6e10ae01f" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44673 } }
I20260929 02:00:51.324514 16036 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7486def25e8c4e67b74e71c6e10ae01f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:00:51.324644 16036 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7486def25e8c4e67b74e71c6e10ae01f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7486def25e8c4e67b74e71c6e10ae01f, State: Initialized, Role: FOLLOWER
I20260929 02:00:51.325300 16036 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7486def25e8c4e67b74e71c6e10ae01f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7486def25e8c4e67b74e71c6e10ae01f" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44673 } }
I20260929 02:00:51.325519 16036 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7486def25e8c4e67b74e71c6e10ae01f [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:00:51.325682 16036 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7486def25e8c4e67b74e71c6e10ae01f [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:00:51.325860 16036 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7486def25e8c4e67b74e71c6e10ae01f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:00:51.328284 16036 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7486def25e8c4e67b74e71c6e10ae01f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7486def25e8c4e67b74e71c6e10ae01f" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44673 } }
I20260929 02:00:51.328805 16036 leader_election.cc:304] T 00000000000000000000000000000000 P 7486def25e8c4e67b74e71c6e10ae01f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7486def25e8c4e67b74e71c6e10ae01f; no voters:
I20260929 02:00:51.329559 16036 leader_election.cc:290] T 00000000000000000000000000000000 P 7486def25e8c4e67b74e71c6e10ae01f [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:00:51.329694 16041 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7486def25e8c4e67b74e71c6e10ae01f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:00:51.330896 16041 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7486def25e8c4e67b74e71c6e10ae01f [term 1 LEADER]: Becoming Leader. State: Replica: 7486def25e8c4e67b74e71c6e10ae01f, State: Running, Role: LEADER
I20260929 02:00:51.331523 16041 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7486def25e8c4e67b74e71c6e10ae01f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7486def25e8c4e67b74e71c6e10ae01f" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44673 } }
I20260929 02:00:51.331846 16036 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7486def25e8c4e67b74e71c6e10ae01f [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:00:51.334153 16042 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7486def25e8c4e67b74e71c6e10ae01f [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7486def25e8c4e67b74e71c6e10ae01f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7486def25e8c4e67b74e71c6e10ae01f" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44673 } } }
I20260929 02:00:51.334444 16042 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7486def25e8c4e67b74e71c6e10ae01f [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:51.334996 16043 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7486def25e8c4e67b74e71c6e10ae01f [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7486def25e8c4e67b74e71c6e10ae01f. Latest consensus state: current_term: 1 leader_uuid: "7486def25e8c4e67b74e71c6e10ae01f" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7486def25e8c4e67b74e71c6e10ae01f" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 44673 } } }
I20260929 02:00:51.335350 16043 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7486def25e8c4e67b74e71c6e10ae01f [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:51.335626 16047 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 02:00:51.346092 16047 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 02:00:51.358215 16047 catalog_manager.cc:1396] Generated new cluster ID: ffa2a2b227714be9a89d3dd2b2bf35e4
I20260929 02:00:51.358342 16047 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 02:00:51.397099 16047 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 02:00:51.398852 16047 catalog_manager.cc:1553] Loading token signing keys...
I20260929 02:00:51.412581 16047 catalog_manager.cc:6219] T 00000000000000000000000000000000 P 7486def25e8c4e67b74e71c6e10ae01f: Generated new TSK 0
I20260929 02:00:51.413620 16047 catalog_manager.cc:1563] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:51.545753 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:44673
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:46549
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:51.880697 16065 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:51.880982 16065 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:51.881080 16065 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:51.889398 16065 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:51.889614 16065 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 02:00:51.900115 16065 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:46549
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:44673
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:51.901659 16065 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:51.902889 16065 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:51.912281 16070 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:51.912247 16071 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:51.913357 16073 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:51.913789 16065 server_base.cc:1061] running on GCE node
I20260929 02:00:51.914355 16065 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:51.915242 16065 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:51.916512 16065 hybrid_clock.cc:648] HybridClock initialized: now 1790647251916421 us; error 39 us; skew 500 ppm
I20260929 02:00:51.916821 16065 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:51 dist-test-slave-vz23 krb5kdc[15953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:51.919767 16065 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 02:00:51.921425 16065 webserver.cc:533] Webserver started at http://127.0.71.65:43791/ using document root <none> and password file <none>
I20260929 02:00:51.922335 16065 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:51.922506 16065 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:51.922871 16065 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:51.925390 16065 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "ac3cd2739ab74ced9c90b718f9f08bd1"
format_stamp: "Formatted at 2026-09-29 02:00:51 on dist-test-slave-vz23"
I20260929 02:00:51.926177 16065 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "ac3cd2739ab74ced9c90b718f9f08bd1"
format_stamp: "Formatted at 2026-09-29 02:00:51 on dist-test-slave-vz23"
I20260929 02:00:51.931480 16065 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:00:51.934841 16080 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:51.936247 16065 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.001s sys 0.001s
I20260929 02:00:51.936429 16065 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "ac3cd2739ab74ced9c90b718f9f08bd1"
format_stamp: "Formatted at 2026-09-29 02:00:51 on dist-test-slave-vz23"
I20260929 02:00:51.936647 16065 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:51.971088 16065 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:51.975240 16065 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:51.975550 16065 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:51.976550 16065 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 02:00:51.978225 16065 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:00:51.978325 16065 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:51.978479 16065 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 02:00:51.978551 16065 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:52.013808 16065 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:41929
I20260929 02:00:52.013860 16192 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:41929 every 8 connection(s)
I20260929 02:00:52.015132 16065 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 02:00:52.021564 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 16065
I20260929 02:00:52.021756 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_9.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
Sep 29 02:00:52 dist-test-slave-vz23 krb5kdc[15953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647251, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 02:00:52.035277 16193 heartbeater.cc:344] Connected to a master server at 127.0.71.126:44673
I20260929 02:00:52.035635 16193 heartbeater.cc:461] Registering TS with master...
I20260929 02:00:52.036640 16193 heartbeater.cc:507] Master 127.0.71.126:44673 requested a full tablet report, sending...
I20260929 02:00:52.041258 16001 ts_manager.cc:194] Registered new tserver with Master: ac3cd2739ab74ced9c90b718f9f08bd1 (127.0.71.65:41929)
I20260929 02:00:52.042944 16001 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:46393
I20260929 02:00:52.049296 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Sep 29 02:00:52 dist-test-slave-vz23 krb5kdc[15953](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:00:52 dist-test-slave-vz23 krb5kdc[15953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.0.71.126@KRBTEST.COM
I20260929 02:00:52.103703 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 16065
Sep 29 02:00:52 dist-test-slave-vz23 krb5kdc[15953](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647252, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 02:00:52.120743 16188 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:00:52.208174 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 16065
I20260929 02:00:52.225178 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 15969
I20260929 02:00:52.226312 16031 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:00:52.335412 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 15969
2026-09-29T02:00:52Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/9 (3945 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/10
Loading random data
Initializing database '/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:00:52 dist-test-slave-vz23 krb5kdc[16206](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:00:52 dist-test-slave-vz23 krb5kdc[16206](info): set up 2 sockets
Sep 29 02:00:52 dist-test-slave-vz23 krb5kdc[16206](info): commencing operation
krb5kdc: starting...
W20260929 02:00:54.465060 285 mini_kdc.cc:121] Time spent starting KDC: real 2.090s user 0.007s sys 0.021s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 02:00:54 dist-test-slave-vz23 krb5kdc[16206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:00:54Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:00:54Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:54.792770 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:33367
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:44243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:33367
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:55.142325 16222 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:55.142679 16222 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:55.142843 16222 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:55.152192 16222 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:00:55.152323 16222 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:55.152487 16222 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:00:55.152597 16222 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:00:55.163856 16222 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:44243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:33367
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:33367
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:55.165612 16222 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:55.167062 16222 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:55.175951 16228 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:55.175946 16227 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:55.176710 16230 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:55.177170 16222 server_base.cc:1061] running on GCE node
I20260929 02:00:55.177994 16222 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:55.179500 16222 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:55.180775 16222 hybrid_clock.cc:648] HybridClock initialized: now 1790647255180722 us; error 57 us; skew 500 ppm
I20260929 02:00:55.181222 16222 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:55 dist-test-slave-vz23 krb5kdc[16206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:55.184417 16222 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 02:00:55.186255 16222 webserver.cc:533] Webserver started at http://127.0.71.126:32845/ using document root <none> and password file <none>
I20260929 02:00:55.187039 16222 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:55.187189 16222 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:55.187558 16222 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:55.190191 16222 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "46d5afffe13f4a74a5987ba7fb5baf34"
format_stamp: "Formatted at 2026-09-29 02:00:55 on dist-test-slave-vz23"
I20260929 02:00:55.191039 16222 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "46d5afffe13f4a74a5987ba7fb5baf34"
format_stamp: "Formatted at 2026-09-29 02:00:55 on dist-test-slave-vz23"
I20260929 02:00:55.196981 16222 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.003s sys 0.004s
I20260929 02:00:55.200760 16237 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:55.202184 16222 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:00:55.202392 16222 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "46d5afffe13f4a74a5987ba7fb5baf34"
format_stamp: "Formatted at 2026-09-29 02:00:55 on dist-test-slave-vz23"
I20260929 02:00:55.202606 16222 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:55.221813 16222 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:55.225690 16222 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:55.225997 16222 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:55.246703 16222 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:33367
I20260929 02:00:55.246734 16288 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:33367 every 8 connection(s)
I20260929 02:00:55.248217 16222 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 02:00:55.249073 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 16222
I20260929 02:00:55.249253 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 02:00:55.253175 16289 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
Sep 29 02:00:55 dist-test-slave-vz23 krb5kdc[16206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647254, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 02:00:55.263089 16289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46d5afffe13f4a74a5987ba7fb5baf34: Bootstrap starting.
I20260929 02:00:55.266731 16289 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 46d5afffe13f4a74a5987ba7fb5baf34: Neither blocks nor log segments found. Creating new log.
I20260929 02:00:55.267812 16289 log.cc:826] T 00000000000000000000000000000000 P 46d5afffe13f4a74a5987ba7fb5baf34: Log is configured to *not* fsync() on all Append() calls
I20260929 02:00:55.271023 16289 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 46d5afffe13f4a74a5987ba7fb5baf34: No bootstrap required, opened a new log
I20260929 02:00:55.276559 16289 raft_consensus.cc:359] T 00000000000000000000000000000000 P 46d5afffe13f4a74a5987ba7fb5baf34 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d5afffe13f4a74a5987ba7fb5baf34" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33367 } }
I20260929 02:00:55.276957 16289 raft_consensus.cc:385] T 00000000000000000000000000000000 P 46d5afffe13f4a74a5987ba7fb5baf34 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:00:55.277074 16289 raft_consensus.cc:740] T 00000000000000000000000000000000 P 46d5afffe13f4a74a5987ba7fb5baf34 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 46d5afffe13f4a74a5987ba7fb5baf34, State: Initialized, Role: FOLLOWER
I20260929 02:00:55.277882 16289 consensus_queue.cc:260] T 00000000000000000000000000000000 P 46d5afffe13f4a74a5987ba7fb5baf34 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d5afffe13f4a74a5987ba7fb5baf34" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33367 } }
I20260929 02:00:55.278115 16289 raft_consensus.cc:399] T 00000000000000000000000000000000 P 46d5afffe13f4a74a5987ba7fb5baf34 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:00:55.278260 16289 raft_consensus.cc:493] T 00000000000000000000000000000000 P 46d5afffe13f4a74a5987ba7fb5baf34 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:00:55.278426 16289 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 46d5afffe13f4a74a5987ba7fb5baf34 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:00:55.280700 16289 raft_consensus.cc:515] T 00000000000000000000000000000000 P 46d5afffe13f4a74a5987ba7fb5baf34 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d5afffe13f4a74a5987ba7fb5baf34" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33367 } }
I20260929 02:00:55.281185 16289 leader_election.cc:304] T 00000000000000000000000000000000 P 46d5afffe13f4a74a5987ba7fb5baf34 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 46d5afffe13f4a74a5987ba7fb5baf34; no voters:
I20260929 02:00:55.281889 16289 leader_election.cc:290] T 00000000000000000000000000000000 P 46d5afffe13f4a74a5987ba7fb5baf34 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:00:55.281944 16294 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 46d5afffe13f4a74a5987ba7fb5baf34 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:00:55.283203 16294 raft_consensus.cc:697] T 00000000000000000000000000000000 P 46d5afffe13f4a74a5987ba7fb5baf34 [term 1 LEADER]: Becoming Leader. State: Replica: 46d5afffe13f4a74a5987ba7fb5baf34, State: Running, Role: LEADER
I20260929 02:00:55.283720 16294 consensus_queue.cc:237] T 00000000000000000000000000000000 P 46d5afffe13f4a74a5987ba7fb5baf34 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d5afffe13f4a74a5987ba7fb5baf34" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33367 } }
I20260929 02:00:55.284140 16289 sys_catalog.cc:565] T 00000000000000000000000000000000 P 46d5afffe13f4a74a5987ba7fb5baf34 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:00:55.286823 16296 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46d5afffe13f4a74a5987ba7fb5baf34 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 46d5afffe13f4a74a5987ba7fb5baf34. Latest consensus state: current_term: 1 leader_uuid: "46d5afffe13f4a74a5987ba7fb5baf34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d5afffe13f4a74a5987ba7fb5baf34" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33367 } } }
I20260929 02:00:55.287216 16296 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46d5afffe13f4a74a5987ba7fb5baf34 [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:55.287438 16295 sys_catalog.cc:455] T 00000000000000000000000000000000 P 46d5afffe13f4a74a5987ba7fb5baf34 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "46d5afffe13f4a74a5987ba7fb5baf34" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "46d5afffe13f4a74a5987ba7fb5baf34" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 33367 } } }
I20260929 02:00:55.287788 16295 sys_catalog.cc:458] T 00000000000000000000000000000000 P 46d5afffe13f4a74a5987ba7fb5baf34 [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:55.290256 16303 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 02:00:55.294838 16303 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 02:00:55.303356 16303 catalog_manager.cc:1396] Generated new cluster ID: 2b82f9813e914da1830bd7fe104f9d7c
I20260929 02:00:55.303469 16303 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 02:00:55.322715 16303 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 02:00:55.323935 16303 catalog_manager.cc:1553] Loading token signing keys...
I20260929 02:00:55.334762 16303 catalog_manager.cc:6219] T 00000000000000000000000000000000 P 46d5afffe13f4a74a5987ba7fb5baf34: Generated new TSK 0
I20260929 02:00:55.335659 16303 catalog_manager.cc:1563] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:55.464044 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:33367
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:44243
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:55.803161 16317 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:55.803472 16317 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:55.803562 16317 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:55.812295 16317 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:55.812628 16317 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 02:00:55.823710 16317 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:44243
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:33367
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:55.825398 16317 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:55.826745 16317 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:55.836690 16323 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:55.837009 16322 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:55.837054 16325 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:55.838543 16317 server_base.cc:1061] running on GCE node
I20260929 02:00:55.839205 16317 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:55.840261 16317 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:55.841496 16317 hybrid_clock.cc:648] HybridClock initialized: now 1790647255841431 us; error 65 us; skew 500 ppm
I20260929 02:00:55.841830 16317 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:55 dist-test-slave-vz23 krb5kdc[16206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:55.845079 16317 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 02:00:55.846830 16317 webserver.cc:533] Webserver started at http://127.0.71.65:36301/ using document root <none> and password file <none>
I20260929 02:00:55.847573 16317 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:55.847712 16317 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:55.848078 16317 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:55.850632 16317 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "7f895e2fda204aacae37baa03e47d34a"
format_stamp: "Formatted at 2026-09-29 02:00:55 on dist-test-slave-vz23"
I20260929 02:00:55.851409 16317 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "7f895e2fda204aacae37baa03e47d34a"
format_stamp: "Formatted at 2026-09-29 02:00:55 on dist-test-slave-vz23"
I20260929 02:00:55.856870 16317 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.003s sys 0.005s
I20260929 02:00:55.860425 16332 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:55.861896 16317 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.004s sys 0.000s
I20260929 02:00:55.862110 16317 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "7f895e2fda204aacae37baa03e47d34a"
format_stamp: "Formatted at 2026-09-29 02:00:55 on dist-test-slave-vz23"
I20260929 02:00:55.862316 16317 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:55.904654 16317 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:55.908794 16317 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:55.909103 16317 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:55.910169 16317 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 02:00:55.911690 16317 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:00:55.911794 16317 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:55.911942 16317 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 02:00:55.912015 16317 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:55.947435 16317 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:37855
I20260929 02:00:55.947525 16444 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:37855 every 8 connection(s)
I20260929 02:00:55.948897 16317 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 02:00:55.951099 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 16317
I20260929 02:00:55.951287 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_10.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
Sep 29 02:00:55 dist-test-slave-vz23 krb5kdc[16206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647255, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 02:00:55.970592 16445 heartbeater.cc:344] Connected to a master server at 127.0.71.126:33367
I20260929 02:00:55.970913 16445 heartbeater.cc:461] Registering TS with master...
I20260929 02:00:55.971752 16445 heartbeater.cc:507] Master 127.0.71.126:33367 requested a full tablet report, sending...
I20260929 02:00:55.973867 16254 ts_manager.cc:194] Registered new tserver with Master: 7f895e2fda204aacae37baa03e47d34a (127.0.71.65:37855)
I20260929 02:00:55.975695 16254 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:48331
I20260929 02:00:55.979362 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
Sep 29 02:00:56 dist-test-slave-vz23 krb5kdc[16206](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:00:56 dist-test-slave-vz23 krb5kdc[16206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.0.71.126@KRBTEST.COM
I20260929 02:00:56.036919 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 16317
Sep 29 02:00:56 dist-test-slave-vz23 krb5kdc[16206](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647256, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 02:00:56.056064 16440 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:00:56.145964 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 16317
I20260929 02:00:56.163472 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 16222
I20260929 02:00:56.164737 16284 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:00:56.271852 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 16222
2026-09-29T02:00:56Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/10 (3937 ms)
[ RUN ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/11
Loading random data
Initializing database '/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/krb5kdc/principal' for realm 'KRBTEST.COM',
master key name 'K/M@KRBTEST.COM'
Sep 29 02:00:56 dist-test-slave-vz23 krb5kdc[16458](info): setting up network...
krb5kdc: setsockopt(10,IPV6_V6ONLY,1) worked
Sep 29 02:00:56 dist-test-slave-vz23 krb5kdc[16458](info): set up 2 sockets
Sep 29 02:00:56 dist-test-slave-vz23 krb5kdc[16458](info): commencing operation
krb5kdc: starting...
W20260929 02:00:58.373093 285 mini_kdc.cc:121] Time spent starting KDC: real 2.060s user 0.000s sys 0.028s
WARNING: no policy specified for test-admin@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-admin@KRBTEST.COM" created.
WARNING: no policy specified for test-user@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "test-user@KRBTEST.COM" created.
WARNING: no policy specified for joe-interloper@KRBTEST.COM; defaulting to no policy
Authenticating as principal slave/admin@KRBTEST.COM with password.
Principal "joe-interloper@KRBTEST.COM" created.
Authenticating as principal slave/admin@KRBTEST.COM with password.
Entry for principal test-user with kvno 1, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/krb5kdc/test-user.keytab.
Entry for principal test-user with kvno 1, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/krb5kdc/test-user.keytab.
Sep 29 02:00:58 dist-test-slave-vz23 krb5kdc[16458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
2026-09-29T02:00:58Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:00:58Z Disabled control of system clock
WARNING: no policy specified for kudu/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.126@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.126@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.126 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:58.704654 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:45785
--webserver_interface=127.0.71.126
--webserver_port=0
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:45785
--enable_rest_api=true with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:59.042929 16474 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:59.043402 16474 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:59.043578 16474 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:59.058488 16474 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:00:59.058702 16474 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:59.058804 16474 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:00:59.058878 16474 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
I20260929 02:00:59.075982 16474 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/master-0/wal
--enable_rest_api=true
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:45785
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.126
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:45785
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--webserver_require_spnego=true
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:59.078256 16474 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:59.080103 16474 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:59.088641 16479 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:59.089013 16480 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:59.090209 16482 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:59.091228 16474 server_base.cc:1061] running on GCE node
I20260929 02:00:59.091909 16474 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:59.093333 16474 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:59.094535 16474 hybrid_clock.cc:648] HybridClock initialized: now 1790647259094487 us; error 35 us; skew 500 ppm
I20260929 02:00:59.094882 16474 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:59 dist-test-slave-vz23 krb5kdc[16458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.126@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:59.098040 16474 init.cc:377] Logged in from keytab as kudu/127.0.71.126@KRBTEST.COM (short username kudu)
I20260929 02:00:59.099725 16474 webserver.cc:533] Webserver started at http://127.0.71.126:32817/ using document root <none> and password file <none>
I20260929 02:00:59.100543 16474 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:59.100663 16474 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:59.101012 16474 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:59.103437 16474 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "7101cf3de82d4bb7abc1735d48ae4212"
format_stamp: "Formatted at 2026-09-29 02:00:59 on dist-test-slave-vz23"
I20260929 02:00:59.104197 16474 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "7101cf3de82d4bb7abc1735d48ae4212"
format_stamp: "Formatted at 2026-09-29 02:00:59 on dist-test-slave-vz23"
I20260929 02:00:59.110401 16474 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.002s sys 0.003s
I20260929 02:00:59.114816 16489 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:59.116425 16474 fs_manager.cc:730] Time spent opening block manager: real 0.004s user 0.002s sys 0.000s
I20260929 02:00:59.116791 16474 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "7101cf3de82d4bb7abc1735d48ae4212"
format_stamp: "Formatted at 2026-09-29 02:00:59 on dist-test-slave-vz23"
I20260929 02:00:59.117023 16474 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:59.135119 16474 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:59.139034 16474 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:59.139340 16474 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:59.159795 16474 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:45785
I20260929 02:00:59.159834 16540 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:45785 every 8 connection(s)
I20260929 02:00:59.161319 16474 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 02:00:59.165300 16541 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:00:59.171337 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 16474
I20260929 02:00:59.171530 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 02:00:59.175462 16541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7101cf3de82d4bb7abc1735d48ae4212: Bootstrap starting.
I20260929 02:00:59.178615 16541 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 7101cf3de82d4bb7abc1735d48ae4212: Neither blocks nor log segments found. Creating new log.
I20260929 02:00:59.179761 16541 log.cc:826] T 00000000000000000000000000000000 P 7101cf3de82d4bb7abc1735d48ae4212: Log is configured to *not* fsync() on all Append() calls
I20260929 02:00:59.182734 16541 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 7101cf3de82d4bb7abc1735d48ae4212: No bootstrap required, opened a new log
Sep 29 02:00:59 dist-test-slave-vz23 krb5kdc[16458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647258, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 02:00:59.189237 16541 raft_consensus.cc:359] T 00000000000000000000000000000000 P 7101cf3de82d4bb7abc1735d48ae4212 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7101cf3de82d4bb7abc1735d48ae4212" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 45785 } }
I20260929 02:00:59.189667 16541 raft_consensus.cc:385] T 00000000000000000000000000000000 P 7101cf3de82d4bb7abc1735d48ae4212 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:00:59.189797 16541 raft_consensus.cc:740] T 00000000000000000000000000000000 P 7101cf3de82d4bb7abc1735d48ae4212 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 7101cf3de82d4bb7abc1735d48ae4212, State: Initialized, Role: FOLLOWER
I20260929 02:00:59.190498 16541 consensus_queue.cc:260] T 00000000000000000000000000000000 P 7101cf3de82d4bb7abc1735d48ae4212 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7101cf3de82d4bb7abc1735d48ae4212" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 45785 } }
I20260929 02:00:59.190739 16541 raft_consensus.cc:399] T 00000000000000000000000000000000 P 7101cf3de82d4bb7abc1735d48ae4212 [term 0 FOLLOWER]: Only one voter in the Raft config. Triggering election immediately
I20260929 02:00:59.190883 16541 raft_consensus.cc:493] T 00000000000000000000000000000000 P 7101cf3de82d4bb7abc1735d48ae4212 [term 0 FOLLOWER]: Starting leader election (initial election of a single-replica configuration)
I20260929 02:00:59.191082 16541 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 7101cf3de82d4bb7abc1735d48ae4212 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:00:59.193816 16541 raft_consensus.cc:515] T 00000000000000000000000000000000 P 7101cf3de82d4bb7abc1735d48ae4212 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7101cf3de82d4bb7abc1735d48ae4212" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 45785 } }
I20260929 02:00:59.194299 16541 leader_election.cc:304] T 00000000000000000000000000000000 P 7101cf3de82d4bb7abc1735d48ae4212 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 1 responses out of 1 voters: 1 yes votes; 0 no votes. yes voters: 7101cf3de82d4bb7abc1735d48ae4212; no voters:
I20260929 02:00:59.194942 16541 leader_election.cc:290] T 00000000000000000000000000000000 P 7101cf3de82d4bb7abc1735d48ae4212 [CANDIDATE]: Term 1 election: Requested vote from peers
I20260929 02:00:59.195117 16546 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 7101cf3de82d4bb7abc1735d48ae4212 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:00:59.195945 16546 raft_consensus.cc:697] T 00000000000000000000000000000000 P 7101cf3de82d4bb7abc1735d48ae4212 [term 1 LEADER]: Becoming Leader. State: Replica: 7101cf3de82d4bb7abc1735d48ae4212, State: Running, Role: LEADER
I20260929 02:00:59.196607 16546 consensus_queue.cc:237] T 00000000000000000000000000000000 P 7101cf3de82d4bb7abc1735d48ae4212 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 1, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7101cf3de82d4bb7abc1735d48ae4212" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 45785 } }
I20260929 02:00:59.198215 16541 sys_catalog.cc:565] T 00000000000000000000000000000000 P 7101cf3de82d4bb7abc1735d48ae4212 [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:00:59.200063 16547 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7101cf3de82d4bb7abc1735d48ae4212 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 7101cf3de82d4bb7abc1735d48ae4212. Latest consensus state: current_term: 1 leader_uuid: "7101cf3de82d4bb7abc1735d48ae4212" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7101cf3de82d4bb7abc1735d48ae4212" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 45785 } } }
I20260929 02:00:59.200070 16548 sys_catalog.cc:455] T 00000000000000000000000000000000 P 7101cf3de82d4bb7abc1735d48ae4212 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 1 leader_uuid: "7101cf3de82d4bb7abc1735d48ae4212" committed_config { opid_index: -1 OBSOLETE_local: true peers { permanent_uuid: "7101cf3de82d4bb7abc1735d48ae4212" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 45785 } } }
I20260929 02:00:59.206235 16548 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7101cf3de82d4bb7abc1735d48ae4212 [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:59.206054 16547 sys_catalog.cc:458] T 00000000000000000000000000000000 P 7101cf3de82d4bb7abc1735d48ae4212 [sys.catalog]: This master's current role is: LEADER
I20260929 02:00:59.208374 16554 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 02:00:59.214041 16554 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 02:00:59.223004 16554 catalog_manager.cc:1396] Generated new cluster ID: a1288d5a712343f286a39919dd3f347d
I20260929 02:00:59.223129 16554 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 02:00:59.249589 16554 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 02:00:59.251498 16554 catalog_manager.cc:1553] Loading token signing keys...
I20260929 02:00:59.267786 16554 catalog_manager.cc:6219] T 00000000000000000000000000000000 P 7101cf3de82d4bb7abc1735d48ae4212: Generated new TSK 0
I20260929 02:00:59.268989 16554 catalog_manager.cc:1563] Initializing in-progress tserver states...
WARNING: no policy specified for kudu/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "kudu/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal kudu/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
WARNING: no policy specified for HTTP/127.0.71.65@KRBTEST.COM; defaulting to no policy
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Principal "HTTP/127.0.71.65@KRBTEST.COM" created.
Authenticating as principal test-admin/admin@KRBTEST.COM with password.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes256-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
Entry for principal HTTP/127.0.71.65 with kvno 2, encryption type aes128-cts-hmac-sha1-96 added to keytab WRFILE:/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/krb5kdc/kudu.keytab.
I20260929 02:00:59.396934 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:45785
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--rpc_authentication=required
--superuser_acl=test-admin
--user_acl=test-user
--webserver_require_spnego=true
--builtin_ntp_servers=127.0.71.84:34043
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {KRB5CCNAME=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/krb5kdc/krb5cc,KRB5_CONFIG=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/krb5kdc/krb5.conf,KRB5_KDC_PROFILE=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/krb5kdc/kdc.conf,KUDU_ENABLE_KRB5_REALM_FIX=yes}
W20260929 02:00:59.739609 16569 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:00:59.739902 16569 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:00:59.740101 16569 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:00:59.748654 16569 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:00:59.748875 16569 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 02:00:59.759318 16569 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:34043
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/ts-0/wal
--keytab_file=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/krb5kdc/kudu.keytab
--principal=kudu/127.0.71.65
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--rpc_authentication=required
--superuser_acl=<redacted>
--user_acl=<redacted>
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--webserver_require_spnego=true
--tserver_master_addrs=127.0.71.126:45785
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:00:59.760905 16569 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:00:59.762331 16569 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:00:59.772168 16575 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:59.772539 16574 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:00:59.772672 16577 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:00:59.774268 16569 server_base.cc:1061] running on GCE node
I20260929 02:00:59.774828 16569 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:00:59.775846 16569 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:00:59.777092 16569 hybrid_clock.cc:648] HybridClock initialized: now 1790647259777047 us; error 53 us; skew 500 ppm
I20260929 02:00:59.777436 16569 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
Sep 29 02:00:59 dist-test-slave-vz23 krb5kdc[16458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
I20260929 02:00:59.780627 16569 init.cc:377] Logged in from keytab as kudu/127.0.71.65@KRBTEST.COM (short username kudu)
I20260929 02:00:59.782531 16569 webserver.cc:533] Webserver started at http://127.0.71.65:42769/ using document root <none> and password file <none>
I20260929 02:00:59.783209 16569 fs_manager.cc:362] Metadata directory not provided
I20260929 02:00:59.783298 16569 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:00:59.783582 16569 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:00:59.785983 16569 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "62666267c88c468e8fbadda659f86bbd"
format_stamp: "Formatted at 2026-09-29 02:00:59 on dist-test-slave-vz23"
I20260929 02:00:59.786738 16569 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "62666267c88c468e8fbadda659f86bbd"
format_stamp: "Formatted at 2026-09-29 02:00:59 on dist-test-slave-vz23"
I20260929 02:00:59.791814 16569 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 02:00:59.795221 16584 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:59.796752 16569 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:00:59.796932 16569 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "62666267c88c468e8fbadda659f86bbd"
format_stamp: "Formatted at 2026-09-29 02:00:59 on dist-test-slave-vz23"
I20260929 02:00:59.797122 16569 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:00:59.825316 16569 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:00:59.829366 16569 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:00:59.829679 16569 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:00:59.830626 16569 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 02:00:59.832125 16569 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:00:59.832199 16569 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:59.832321 16569 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 02:00:59.832387 16569 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:00:59.867112 16569 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:35917
I20260929 02:00:59.867193 16696 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:35917 every 8 connection(s)
I20260929 02:00:59.868531 16569 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 02:00:59.877914 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 16569
I20260929 02:00:59.878100 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl_11.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
Sep 29 02:00:59 dist-test-slave-vz23 krb5kdc[16458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647259, etypes {rep=17 tkt=17 ses=17}, kudu/127.0.71.65@KRBTEST.COM for kudu/127.0.71.126@KRBTEST.COM
I20260929 02:00:59.889004 16697 heartbeater.cc:344] Connected to a master server at 127.0.71.126:45785
I20260929 02:00:59.889360 16697 heartbeater.cc:461] Registering TS with master...
I20260929 02:00:59.890151 16697 heartbeater.cc:507] Master 127.0.71.126:45785 requested a full tablet report, sending...
I20260929 02:00:59.892391 16506 ts_manager.cc:194] Registered new tserver with Master: 62666267c88c468e8fbadda659f86bbd (127.0.71.65:35917)
I20260929 02:00:59.894119 285 external_mini_cluster.cc:955] 1 TS(s) registered with all masters
I20260929 02:00:59.894279 16506 master_service.cc:502] Signed X509 certificate for tserver {username='kudu', principal='kudu/127.0.71.65@KRBTEST.COM'} at 127.0.71.65:38175
Sep 29 02:00:59 dist-test-slave-vz23 krb5kdc[16458](info): AS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for krbtgt/KRBTEST.COM@KRBTEST.COM
Password for test-admin@KRBTEST.COM:
Sep 29 02:00:59 dist-test-slave-vz23 krb5kdc[16458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for HTTP/127.0.71.126@KRBTEST.COM
I20260929 02:00:59.946054 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 16569
Sep 29 02:00:59 dist-test-slave-vz23 krb5kdc[16458](info): TGS_REQ (2 etypes {17 16}) 127.0.0.1: ISSUE: authtime 1790647259, etypes {rep=17 tkt=17 ses=17}, test-admin@KRBTEST.COM for kudu/127.0.71.65@KRBTEST.COM
I20260929 02:00:59.964174 16692 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:01:00.052758 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 16569
I20260929 02:01:00.070020 285 external_mini_cluster.cc:1705] Attempting to check leaks for /tmp/dist-test-taskniqJIA/build/asan/bin/kudu pid 16474
I20260929 02:01:00.071226 16536 generic_service.cc:196] Checking for leaks (request via RPC)
I20260929 02:01:00.179733 285 external_mini_cluster.cc:1664] Killing /tmp/dist-test-taskniqJIA/build/asan/bin/kudu with pid 16474
2026-09-29T02:01:00Z chronyd exiting
[ OK ] ExternalMiniClusterTest.TestRestApiSpnegoConnectionThroughCurl/11 (3907 ms)
[----------] 37 tests from ExternalMiniClusterTest (656788 ms total)
[----------] 1 test from Kudu3762Test
[ RUN ] Kudu3762Test.AddMaster
2026-09-29T02:01:00Z chronyd version 4.6.1 starting (+CMDMON +NTP +REFCLOCK +RTC -PRIVDROP -SCFILTER -SIGND +ASYNCDNS -NTS -SECHASH -IPV6 +DEBUG)
2026-09-29T02:01:00Z Disabled control of system clock
I20260929 02:01:00.259218 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:39181
--webserver_interface=127.0.71.126
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:39181,127.0.71.125:36751
--flush_threshold_mb=0
--flush_threshold_secs=1
--log_cache_size_limit_mb=1
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_segment_size_mb=1
--maintenance_manager_num_threads=4
--maintenance_manager_polling_interval_ms=10
--unlock_experimental_flags=true with env {}
W20260929 02:01:00.595031 16711 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:01:00.595373 16711 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:01:00.595455 16711 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:01:00.603984 16711 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20260929 02:01:00.604086 16711 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20260929 02:01:00.604156 16711 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:01:00.604197 16711 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:01:00.604264 16711 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:01:00.604311 16711 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260929 02:01:00.604396 16711 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260929 02:01:00.604454 16711 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
I20260929 02:01:00.614784 16711 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-0/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:39181,127.0.71.125:36751
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.126:39181
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
--webserver_interface=127.0.71.126
--webserver_port=0
--flush_threshold_mb=0
--flush_threshold_secs=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_num_threads=4
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
W20260929 02:01:00.615136 16711 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.71.126:39181,127.0.71.125:36751'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 02:01:00.616899 16711 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:01:00.619174 16711 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:01:00.627496 16717 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:01:00.627521 16716 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:01:00.628718 16711 server_base.cc:1061] running on GCE node
W20260929 02:01:00.629634 16719 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:01:00.630405 16711 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:01:00.631825 16711 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:01:00.633147 16711 hybrid_clock.cc:648] HybridClock initialized: now 1790647260633076 us; error 61 us; skew 500 ppm
I20260929 02:01:00.633515 16711 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:01:00.635653 16711 webserver.cc:533] Webserver started at http://127.0.71.126:36965/ using document root <none> and password file <none>
I20260929 02:01:00.636338 16711 fs_manager.cc:362] Metadata directory not provided
I20260929 02:01:00.636520 16711 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:01:00.636863 16711 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:01:00.639174 16711 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-0/data/instance:
uuid: "463404b2470c42c9b6bd67f027cf5e2b"
format_stamp: "Formatted at 2026-09-29 02:01:00 on dist-test-slave-vz23"
I20260929 02:01:00.639909 16711 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-0/wal/instance:
uuid: "463404b2470c42c9b6bd67f027cf5e2b"
format_stamp: "Formatted at 2026-09-29 02:01:00 on dist-test-slave-vz23"
I20260929 02:01:00.645187 16711 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.003s
I20260929 02:01:00.648669 16725 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:01:00.650002 16711 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.001s
I20260929 02:01:00.650179 16711 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-0/wal
uuid: "463404b2470c42c9b6bd67f027cf5e2b"
format_stamp: "Formatted at 2026-09-29 02:01:00 on dist-test-slave-vz23"
I20260929 02:01:00.650378 16711 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:01:00.670568 16711 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:01:00.671439 16711 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:01:00.671739 16711 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:01:00.693728 16711 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.126:39181
I20260929 02:01:00.693759 16779 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.126:39181 every 8 connection(s)
I20260929 02:01:00.695302 16711 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-0/data/info.pb
I20260929 02:01:00.699476 16780 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:00.704807 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 16711
I20260929 02:01:00.705065 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-0/wal/instance
I20260929 02:01:00.705327 16780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39181 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:01:00.706347 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
master
run
--ipki_ca_key_size=768
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:36751
--webserver_interface=127.0.71.125
--webserver_port=0
--builtin_ntp_servers=127.0.71.84:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--rpc_reuseport=true
--master_addresses=127.0.71.126:39181,127.0.71.125:36751
--flush_threshold_mb=0
--flush_threshold_secs=1
--log_cache_size_limit_mb=1
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_segment_size_mb=1
--maintenance_manager_num_threads=4
--maintenance_manager_polling_interval_ms=10
--unlock_experimental_flags=true with env {}
I20260929 02:01:00.720649 16780 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36751 } has no permanent_uuid. Determining permanent_uuid...
W20260929 02:01:00.722118 16727 proxy.cc:239] Call had error, refreshing address and retrying: Network error: Client connection negotiation failed: client connection to 127.0.71.125:36751: connect: Connection refused (error 111)
W20260929 02:01:00.723734 16780 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:36751: Network error: Client connection negotiation failed: client connection to 127.0.71.125:36751: connect: Connection refused (error 111)
I20260929 02:01:00.773089 16780 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36751 } attempt: 1
W20260929 02:01:00.775159 16780 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:36751: Network error: Client connection negotiation failed: client connection to 127.0.71.125:36751: connect: Connection refused (error 111)
I20260929 02:01:00.843433 16780 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36751 } attempt: 2
W20260929 02:01:00.845764 16780 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:36751: Network error: Client connection negotiation failed: client connection to 127.0.71.125:36751: connect: Connection refused (error 111)
I20260929 02:01:00.937141 16780 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36751 } attempt: 3
W20260929 02:01:00.939317 16780 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:36751: Network error: Client connection negotiation failed: client connection to 127.0.71.125:36751: connect: Connection refused (error 111)
W20260929 02:01:01.048157 16784 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:01:01.048527 16784 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:01:01.048689 16784 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:01:01.057564 16784 flags.cc:432] Enabled experimental flag: --log_compression_codec=no_compression
W20260929 02:01:01.057690 16784 flags.cc:432] Enabled experimental flag: --log_max_segments_to_retain=3
W20260929 02:01:01.057824 16784 flags.cc:432] Enabled experimental flag: --ipki_ca_key_size=768
W20260929 02:01:01.057911 16784 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:01:01.058036 16784 flags.cc:432] Enabled experimental flag: --tsk_num_rsa_bits=512
W20260929 02:01:01.058125 16784 flags.cc:432] Enabled experimental flag: --rpc_reuseport=true
W20260929 02:01:01.058261 16784 flags.cc:432] Enabled experimental flag: --flush_threshold_mb=0
W20260929 02:01:01.058336 16784 flags.cc:432] Enabled experimental flag: --flush_threshold_secs=1
I20260929 02:01:01.068814 16784 master_runner.cc:387] Master server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--log_compression_codec=no_compression
--log_max_segments_to_retain=3
--log_cache_size_limit_mb=1
--log_segment_size_mb=1
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-1/wal
--ipki_ca_key_size=768
--master_addresses=127.0.71.126:39181,127.0.71.125:36751
--ipki_server_key_size=768
--openssl_security_level_override=0
--tsk_num_rsa_bits=512
--rpc_bind_addresses=127.0.71.125:36751
--rpc_reuseport=true
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
--webserver_interface=127.0.71.125
--webserver_port=0
--flush_threshold_mb=0
--flush_threshold_secs=1
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--maintenance_manager_num_threads=4
--maintenance_manager_polling_interval_ms=10
--enable_minidumps=false
Master server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
W20260929 02:01:01.069202 16784 master_options.cc:55] Only 2 masters are specified by master_addresses_flag ('127.0.71.126:39181,127.0.71.125:36751'), but minimum of 3 are required to tolerate failures of any one master. It is recommended to use at least 3 masters.
I20260929 02:01:01.070565 16784 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:01:01.072355 16784 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:01:01.080464 16790 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:01:01.080986 16791 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:01:01.081848 16793 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:01:01.082568 16780 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36751 } attempt: 4
I20260929 02:01:01.082758 16784 server_base.cc:1061] running on GCE node
I20260929 02:01:01.083684 16784 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
W20260929 02:01:01.084981 16780 consensus_peers.cc:760] Error getting permanent uuid from config peer 127.0.71.125:36751: Network error: Client connection negotiation failed: client connection to 127.0.71.125:36751: connect: Connection refused (error 111)
I20260929 02:01:01.085405 16784 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:01:01.086615 16784 hybrid_clock.cc:648] HybridClock initialized: now 1790647261086537 us; error 55 us; skew 500 ppm
I20260929 02:01:01.086951 16784 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:01:01.089182 16784 webserver.cc:533] Webserver started at http://127.0.71.125:35295/ using document root <none> and password file <none>
I20260929 02:01:01.089923 16784 fs_manager.cc:362] Metadata directory not provided
I20260929 02:01:01.090056 16784 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:01:01.090404 16784 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:01:01.093406 16784 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-1/data/instance:
uuid: "02c2326bb7f24f5c98809afcc5a36d25"
format_stamp: "Formatted at 2026-09-29 02:01:01 on dist-test-slave-vz23"
I20260929 02:01:01.094475 16784 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-1/wal/instance:
uuid: "02c2326bb7f24f5c98809afcc5a36d25"
format_stamp: "Formatted at 2026-09-29 02:01:01 on dist-test-slave-vz23"
I20260929 02:01:01.100313 16784 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.006s sys 0.000s
I20260929 02:01:01.103981 16799 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:01:01.105504 16784 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:01:01.105701 16784 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-1/wal
uuid: "02c2326bb7f24f5c98809afcc5a36d25"
format_stamp: "Formatted at 2026-09-29 02:01:01 on dist-test-slave-vz23"
I20260929 02:01:01.105997 16784 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:01:01.132790 16784 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:01:01.133698 16784 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:01:01.134001 16784 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:01:01.155480 16784 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.125:36751
I20260929 02:01:01.155504 16853 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.125:36751 every 8 connection(s)
I20260929 02:01:01.157155 16784 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-1/data/info.pb
I20260929 02:01:01.161568 16854 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00000000000000000000000000000000. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:01.165194 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 16784
I20260929 02:01:01.165380 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/master-1/wal/instance
I20260929 02:01:01.168023 16854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39181 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:01:01.182212 16854 sys_catalog.cc:422] member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36751 } has no permanent_uuid. Determining permanent_uuid...
I20260929 02:01:01.193077 16854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Bootstrap starting.
I20260929 02:01:01.196437 16854 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:01.197479 16854 log.cc:826] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Log is configured to *not* fsync() on all Append() calls
I20260929 02:01:01.200173 16854 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: No bootstrap required, opened a new log
I20260929 02:01:01.205121 16854 raft_consensus.cc:359] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "463404b2470c42c9b6bd67f027cf5e2b" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39181 } } peers { permanent_uuid: "02c2326bb7f24f5c98809afcc5a36d25" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36751 } }
I20260929 02:01:01.205396 16854 raft_consensus.cc:385] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:01.205478 16854 raft_consensus.cc:740] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 02c2326bb7f24f5c98809afcc5a36d25, State: Initialized, Role: FOLLOWER
I20260929 02:01:01.206009 16854 consensus_queue.cc:260] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "463404b2470c42c9b6bd67f027cf5e2b" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39181 } } peers { permanent_uuid: "02c2326bb7f24f5c98809afcc5a36d25" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36751 } }
I20260929 02:01:01.206717 16861 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25 [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "463404b2470c42c9b6bd67f027cf5e2b" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39181 } } peers { permanent_uuid: "02c2326bb7f24f5c98809afcc5a36d25" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36751 } } }
I20260929 02:01:01.206970 16861 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25 [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:01:01.207804 16854 sys_catalog.cc:565] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25 [sys.catalog]: configured and running, proceeding with master startup.
W20260929 02:01:01.208381 16808 tablet.cc:2433] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Can't schedule compaction. Clean time has not been advanced past its initial value.
W20260929 02:01:01.218083 16872 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:01:01.218214 16872 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:01:01.216220 16784 master_runner.cc:187] Error getting master registration for 127.0.71.126:39181: OK, instance_id { permanent_uuid: "463404b2470c42c9b6bd67f027cf5e2b" instance_seqno: 1790647260688056 } error { code: CATALOG_MANAGER_NOT_INITIALIZED status { code: SERVICE_UNAVAILABLE message: "Catalog manager is not initialized. State: Starting" } }
I20260929 02:01:01.221140 16784 master_runner.cc:419] Couldn't verify the masters in the cluster. Trying again...
I20260929 02:01:01.383602 16861 raft_consensus.cc:493] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:01.383893 16861 raft_consensus.cc:515] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "463404b2470c42c9b6bd67f027cf5e2b" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39181 } } peers { permanent_uuid: "02c2326bb7f24f5c98809afcc5a36d25" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36751 } }
I20260929 02:01:01.384245 16780 consensus_peers.cc:770] Retrying to get permanent uuid for remote peer: member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36751 } attempt: 5
I20260929 02:01:01.385727 16861 leader_election.cc:290] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 463404b2470c42c9b6bd67f027cf5e2b (127.0.71.126:39181)
I20260929 02:01:01.394850 16780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 463404b2470c42c9b6bd67f027cf5e2b: Bootstrap starting.
I20260929 02:01:01.398625 16780 tablet_bootstrap.cc:654] T 00000000000000000000000000000000 P 463404b2470c42c9b6bd67f027cf5e2b: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:01.399710 16780 log.cc:826] T 00000000000000000000000000000000 P 463404b2470c42c9b6bd67f027cf5e2b: Log is configured to *not* fsync() on all Append() calls
I20260929 02:01:01.402979 16780 tablet_bootstrap.cc:492] T 00000000000000000000000000000000 P 463404b2470c42c9b6bd67f027cf5e2b: No bootstrap required, opened a new log
I20260929 02:01:01.408237 16780 raft_consensus.cc:359] T 00000000000000000000000000000000 P 463404b2470c42c9b6bd67f027cf5e2b [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "463404b2470c42c9b6bd67f027cf5e2b" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39181 } } peers { permanent_uuid: "02c2326bb7f24f5c98809afcc5a36d25" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36751 } }
I20260929 02:01:01.408633 16780 raft_consensus.cc:385] T 00000000000000000000000000000000 P 463404b2470c42c9b6bd67f027cf5e2b [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:01.408748 16780 raft_consensus.cc:740] T 00000000000000000000000000000000 P 463404b2470c42c9b6bd67f027cf5e2b [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 463404b2470c42c9b6bd67f027cf5e2b, State: Initialized, Role: FOLLOWER
I20260929 02:01:01.409500 16780 consensus_queue.cc:260] T 00000000000000000000000000000000 P 463404b2470c42c9b6bd67f027cf5e2b [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "463404b2470c42c9b6bd67f027cf5e2b" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39181 } } peers { permanent_uuid: "02c2326bb7f24f5c98809afcc5a36d25" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36751 } }
I20260929 02:01:01.410437 16875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 463404b2470c42c9b6bd67f027cf5e2b [sys.catalog]: SysCatalogTable state changed. Reason: RaftConsensus started. Latest consensus state: current_term: 0 committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "463404b2470c42c9b6bd67f027cf5e2b" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39181 } } peers { permanent_uuid: "02c2326bb7f24f5c98809afcc5a36d25" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36751 } } }
I20260929 02:01:01.410753 16875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 463404b2470c42c9b6bd67f027cf5e2b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:01:01.411798 16780 sys_catalog.cc:565] T 00000000000000000000000000000000 P 463404b2470c42c9b6bd67f027cf5e2b [sys.catalog]: configured and running, proceeding with master startup.
I20260929 02:01:01.411887 16755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02c2326bb7f24f5c98809afcc5a36d25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "463404b2470c42c9b6bd67f027cf5e2b" is_pre_election: true
I20260929 02:01:01.412423 16755 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 463404b2470c42c9b6bd67f027cf5e2b [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 02c2326bb7f24f5c98809afcc5a36d25 in term 0.
I20260929 02:01:01.413192 16800 leader_election.cc:304] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 02c2326bb7f24f5c98809afcc5a36d25, 463404b2470c42c9b6bd67f027cf5e2b; no voters:
I20260929 02:01:01.413636 16861 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:01.413786 16861 raft_consensus.cc:493] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:01.413890 16861 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:01.417323 16861 raft_consensus.cc:515] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "463404b2470c42c9b6bd67f027cf5e2b" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39181 } } peers { permanent_uuid: "02c2326bb7f24f5c98809afcc5a36d25" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36751 } }
W20260929 02:01:01.417744 16734 tablet.cc:2433] T 00000000000000000000000000000000 P 463404b2470c42c9b6bd67f027cf5e2b: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:01:01.417925 16861 leader_election.cc:290] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25 [CANDIDATE]: Term 1 election: Requested vote from peers 463404b2470c42c9b6bd67f027cf5e2b (127.0.71.126:39181)
I20260929 02:01:01.419047 16755 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00000000000000000000000000000000" candidate_uuid: "02c2326bb7f24f5c98809afcc5a36d25" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "463404b2470c42c9b6bd67f027cf5e2b"
I20260929 02:01:01.419289 16755 raft_consensus.cc:3060] T 00000000000000000000000000000000 P 463404b2470c42c9b6bd67f027cf5e2b [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:01.422844 16755 raft_consensus.cc:2468] T 00000000000000000000000000000000 P 463404b2470c42c9b6bd67f027cf5e2b [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 02c2326bb7f24f5c98809afcc5a36d25 in term 1.
I20260929 02:01:01.423473 16800 leader_election.cc:304] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 2 voters: 2 yes votes; 0 no votes. yes voters: 02c2326bb7f24f5c98809afcc5a36d25, 463404b2470c42c9b6bd67f027cf5e2b; no voters:
I20260929 02:01:01.423791 16861 raft_consensus.cc:2804] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:01.432723 16861 raft_consensus.cc:697] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25 [term 1 LEADER]: Becoming Leader. State: Replica: 02c2326bb7f24f5c98809afcc5a36d25, State: Running, Role: LEADER
I20260929 02:01:01.433418 16861 consensus_queue.cc:237] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "463404b2470c42c9b6bd67f027cf5e2b" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39181 } } peers { permanent_uuid: "02c2326bb7f24f5c98809afcc5a36d25" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36751 } }
W20260929 02:01:01.435370 16886 catalog_manager.cc:1607] T 00000000000000000000000000000000 P 463404b2470c42c9b6bd67f027cf5e2b: loading cluster ID for follower catalog manager: Not found: cluster ID entry not found
W20260929 02:01:01.435763 16886 catalog_manager.cc:896] Not found: cluster ID entry not found: failed to prepare follower catalog manager, will retry
I20260929 02:01:01.440981 16887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25 [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02c2326bb7f24f5c98809afcc5a36d25. Latest consensus state: current_term: 1 leader_uuid: "02c2326bb7f24f5c98809afcc5a36d25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "463404b2470c42c9b6bd67f027cf5e2b" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39181 } } peers { permanent_uuid: "02c2326bb7f24f5c98809afcc5a36d25" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36751 } } }
I20260929 02:01:01.441289 16887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25 [sys.catalog]: This master's current role is: LEADER
I20260929 02:01:01.445916 16890 catalog_manager.cc:1524] Loading table and tablet metadata into memory...
I20260929 02:01:01.448346 16890 catalog_manager.cc:1533] Initializing Kudu cluster ID...
I20260929 02:01:01.456694 16755 raft_consensus.cc:1275] T 00000000000000000000000000000000 P 463404b2470c42c9b6bd67f027cf5e2b [term 1 FOLLOWER]: Refusing update from remote peer 02c2326bb7f24f5c98809afcc5a36d25: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 1 index: 2. (index mismatch)
I20260929 02:01:01.457718 16887 consensus_queue.cc:1048] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25 [LEADER]: Connected to new peer: Peer: permanent_uuid: "463404b2470c42c9b6bd67f027cf5e2b" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39181 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:01.479142 16875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 463404b2470c42c9b6bd67f027cf5e2b [sys.catalog]: SysCatalogTable state changed. Reason: New leader 02c2326bb7f24f5c98809afcc5a36d25. Latest consensus state: current_term: 1 leader_uuid: "02c2326bb7f24f5c98809afcc5a36d25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "463404b2470c42c9b6bd67f027cf5e2b" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39181 } } peers { permanent_uuid: "02c2326bb7f24f5c98809afcc5a36d25" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36751 } } }
I20260929 02:01:01.479373 16875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 463404b2470c42c9b6bd67f027cf5e2b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:01:01.481685 16891 mvcc.cc:204] Tried to move back new op lower bound from 7334491182916255744 to 7334491182835662848. Current Snapshot: MvccSnapshot[applied={T|T < 7334491182916255744}]
I20260929 02:01:01.494033 16887 sys_catalog.cc:455] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25 [sys.catalog]: SysCatalogTable state changed. Reason: Peer health change. Latest consensus state: current_term: 1 leader_uuid: "02c2326bb7f24f5c98809afcc5a36d25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "463404b2470c42c9b6bd67f027cf5e2b" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39181 } } peers { permanent_uuid: "02c2326bb7f24f5c98809afcc5a36d25" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36751 } } }
I20260929 02:01:01.494293 16887 sys_catalog.cc:458] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25 [sys.catalog]: This master's current role is: LEADER
I20260929 02:01:01.495132 16890 catalog_manager.cc:1396] Generated new cluster ID: 880bcdb43e214b76927cd2ef4af8291c
I20260929 02:01:01.495245 16890 catalog_manager.cc:1544] Initializing Kudu internal certificate authority...
I20260929 02:01:01.499541 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:01.514549 16875 sys_catalog.cc:455] T 00000000000000000000000000000000 P 463404b2470c42c9b6bd67f027cf5e2b [sys.catalog]: SysCatalogTable state changed. Reason: Replicated consensus-only round. Latest consensus state: current_term: 1 leader_uuid: "02c2326bb7f24f5c98809afcc5a36d25" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "463404b2470c42c9b6bd67f027cf5e2b" member_type: VOTER last_known_addr { host: "127.0.71.126" port: 39181 } } peers { permanent_uuid: "02c2326bb7f24f5c98809afcc5a36d25" member_type: VOTER last_known_addr { host: "127.0.71.125" port: 36751 } } }
I20260929 02:01:01.514757 16875 sys_catalog.cc:458] T 00000000000000000000000000000000 P 463404b2470c42c9b6bd67f027cf5e2b [sys.catalog]: This master's current role is: FOLLOWER
I20260929 02:01:01.519351 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:01.550558 16890 catalog_manager.cc:1419] Generated new certificate authority record
I20260929 02:01:01.552753 16890 catalog_manager.cc:1553] Loading token signing keys...
I20260929 02:01:01.572153 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:01.572501 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 163 bytes on disk
I20260929 02:01:01.572830 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 163 bytes on disk
I20260929 02:01:01.575672 16890 catalog_manager.cc:6219] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Generated new TSK 0
I20260929 02:01:01.570892 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.070s user 0.026s sys 0.008s Metrics: {"bytes_written":4776,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":194,"dirs.run_wall_time_us":911,"drs_written":1,"lbm_read_time_us":105,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1017,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":1}
I20260929 02:01:01.577877 16890 catalog_manager.cc:1563] Initializing in-progress tserver states...
I20260929 02:01:01.578609 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 163 bytes on disk
I20260929 02:01:01.579422 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":79,"lbm_reads_lt_1ms":4}
I20260929 02:01:01.580026 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:01.580520 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:01.589824 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.070s user 0.023s sys 0.007s Metrics: {"bytes_written":4776,"cfile_init":1,"dirs.queue_time_us":1804,"dirs.run_cpu_time_us":205,"dirs.run_wall_time_us":1349,"drs_written":1,"lbm_read_time_us":139,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1522,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":1,"thread_start_us":1822,"threads_started":1}
I20260929 02:01:01.591807 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 163 bytes on disk
I20260929 02:01:01.592182 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 163 bytes on disk
I20260929 02:01:01.592401 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 174 bytes on disk
I20260929 02:01:01.592530 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:01.592746 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 174 bytes on disk
I20260929 02:01:01.592801 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 163 bytes on disk
I20260929 02:01:01.593035 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 174 bytes on disk
I20260929 02:01:01.592924 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":126,"lbm_reads_lt_1ms":4}
I20260929 02:01:01.593833 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:01.593770 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20260929 02:01:01.594050 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:01.594300 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.002s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:01.595425 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.014997
I20260929 02:01:01.596035 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.014997
I20260929 02:01:01.596570 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.014997
I20260929 02:01:01.601884 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.65:0
--local_ip_for_outbound_sockets=127.0.71.65
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:39181,127.0.71.125:36751
--builtin_ntp_servers=127.0.71.84:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 02:01:01.609347 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.016s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:01.613148 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.020s user 0.014s sys 0.000s Metrics: {"bytes_written":5967,"cfile_init":1,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":1390,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":913,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":2}
I20260929 02:01:01.614693 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.014997
I20260929 02:01:01.615321 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.014997
I20260929 02:01:01.615863 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.014997
I20260929 02:01:01.620134 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.047s user 0.004s sys 0.010s Metrics: {"bytes_written":5967,"cfile_init":1,"dirs.queue_time_us":44,"dirs.run_cpu_time_us":180,"dirs.run_wall_time_us":1827,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":936,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":2}
I20260929 02:01:01.625629 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.014997
I20260929 02:01:01.645720 16731 tablet.cc:2133] T 00000000000000000000000000000000 P 463404b2470c42c9b6bd67f027cf5e2b: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260929 02:01:01.646282 16807 tablet.cc:2133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260929 02:01:01.647871 16732 tablet.cc:2133] T 00000000000000000000000000000000 P 463404b2470c42c9b6bd67f027cf5e2b: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260929 02:01:01.651697 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.026s user 0.009s sys 0.012s Metrics: {"dirs.queue_time_us":2775,"dirs.run_cpu_time_us":800,"dirs.run_wall_time_us":4794,"lbm_write_time_us":230,"lbm_writes_lt_1ms":7,"mutex_wait_us":25,"num_input_rowsets":0,"spinlock_wait_cycles":572800,"thread_start_us":8426,"threads_started":1}
I20260929 02:01:01.654254 16804 tablet.cc:2133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260929 02:01:01.656826 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.040s user 0.008s sys 0.005s Metrics: {"dirs.queue_time_us":1889,"dirs.run_cpu_time_us":755,"dirs.run_wall_time_us":4942,"lbm_write_time_us":332,"lbm_writes_lt_1ms":7,"mutex_wait_us":4365,"num_input_rowsets":0,"spinlock_wait_cycles":57088,"thread_start_us":4681,"threads_started":1}
I20260929 02:01:01.663375 16730 tablet.cc:2133] T 00000000000000000000000000000000 P 463404b2470c42c9b6bd67f027cf5e2b: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260929 02:01:01.668486 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.052s user 0.011s sys 0.000s Metrics: {"dirs.queue_time_us":3240,"dirs.run_cpu_time_us":752,"dirs.run_wall_time_us":3403,"lbm_write_time_us":308,"lbm_writes_lt_1ms":7,"mutex_wait_us":2331,"num_input_rowsets":0,"spinlock_wait_cycles":10470656}
I20260929 02:01:01.669323 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.068s user 0.016s sys 0.008s Metrics: {"dirs.queue_time_us":5780,"dirs.run_cpu_time_us":874,"dirs.run_wall_time_us":4076,"lbm_write_time_us":317,"lbm_writes_lt_1ms":7,"mutex_wait_us":8606,"num_input_rowsets":0,"thread_start_us":20957,"threads_started":2}
I20260929 02:01:01.672567 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.076s user 0.018s sys 0.008s Metrics: {"bytes_written":6027,"cfile_cache_hit":12,"cfile_cache_hit_bytes":1553,"cfile_cache_miss":28,"cfile_cache_miss_bytes":2195,"cfile_init":7,"delta_iterators_relevant":4,"dirs.queue_time_us":337,"dirs.run_cpu_time_us":814,"dirs.run_wall_time_us":4596,"drs_written":1,"lbm_read_time_us":1226,"lbm_reads_lt_1ms":56,"lbm_write_time_us":5430,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":26,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":3}
I20260929 02:01:01.673611 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 185 bytes on disk
I20260929 02:01:01.673939 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 185 bytes on disk
I20260929 02:01:01.674218 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 185 bytes on disk
I20260929 02:01:01.675097 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20260929 02:01:01.675513 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:01.675789 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:01.676568 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.072s user 0.018s sys 0.000s Metrics: {"dirs.queue_time_us":6851,"dirs.run_cpu_time_us":904,"dirs.run_wall_time_us":22185,"lbm_write_time_us":260,"lbm_writes_lt_1ms":7,"mutex_wait_us":12764,"num_input_rowsets":0,"spinlock_wait_cycles":7779840,"thread_start_us":22350,"threads_started":2}
I20260929 02:01:01.700269 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 185 bytes on disk
I20260929 02:01:01.700788 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 185 bytes on disk
I20260929 02:01:01.701062 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 185 bytes on disk
I20260929 02:01:01.701694 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":116,"lbm_reads_lt_1ms":4}
I20260929 02:01:01.702116 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:01.702618 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:01.725106 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.109s user 0.040s sys 0.023s Metrics: {"bytes_written":6027,"cfile_cache_hit":12,"cfile_cache_hit_bytes":1553,"cfile_cache_miss":28,"cfile_cache_miss_bytes":2195,"cfile_init":8,"delta_iterators_relevant":4,"dirs.queue_time_us":4131,"dirs.run_cpu_time_us":529,"dirs.run_wall_time_us":4099,"drs_written":1,"lbm_read_time_us":4101,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":59,"lbm_write_time_us":6153,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":26,"mutex_wait_us":40,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":3,"spinlock_wait_cycles":906496,"thread_start_us":33213,"threads_started":1}
W20260929 02:01:02.058722 16897 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:01:02.059039 16897 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:01:02.059113 16897 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:01:02.067570 16897 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:01:02.067757 16897 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.65
I20260929 02:01:02.078387 16897 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.65:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
--webserver_interface=127.0.71.65
--webserver_port=0
--tserver_master_addrs=127.0.71.126:39181,127.0.71.125:36751
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.65
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:01:02.079998 16897 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:01:02.082006 16897 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:01:02.092682 16912 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:01:02.093122 16909 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:01:02.093240 16910 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:01:02.093744 16897 server_base.cc:1061] running on GCE node
I20260929 02:01:02.094291 16897 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:01:02.095248 16897 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:01:02.096639 16897 hybrid_clock.cc:648] HybridClock initialized: now 1790647262096584 us; error 66 us; skew 500 ppm
I20260929 02:01:02.096988 16897 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:01:02.099128 16897 webserver.cc:533] Webserver started at http://127.0.71.65:45157/ using document root <none> and password file <none>
I20260929 02:01:02.099858 16897 fs_manager.cc:362] Metadata directory not provided
I20260929 02:01:02.099948 16897 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:01:02.100293 16897 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:01:02.102808 16897 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/data/instance:
uuid: "97df55a0361841319adfae27c9787b2f"
format_stamp: "Formatted at 2026-09-29 02:01:02 on dist-test-slave-vz23"
I20260929 02:01:02.103597 16897 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/instance:
uuid: "97df55a0361841319adfae27c9787b2f"
format_stamp: "Formatted at 2026-09-29 02:01:02 on dist-test-slave-vz23"
I20260929 02:01:02.108871 16897 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.007s sys 0.000s
I20260929 02:01:02.112293 16918 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:01:02.113723 16897 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:01:02.113903 16897 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal
uuid: "97df55a0361841319adfae27c9787b2f"
format_stamp: "Formatted at 2026-09-29 02:01:02 on dist-test-slave-vz23"
I20260929 02:01:02.114128 16897 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:01:02.139432 16897 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:01:02.140343 16897 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:01:02.140666 16897 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:01:02.141722 16897 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 02:01:02.143404 16897 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:01:02.143543 16897 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:01:02.143666 16897 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 02:01:02.143754 16897 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:01:02.186985 16897 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.65:43625
I20260929 02:01:02.187036 17030 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.65:43625 every 8 connection(s)
I20260929 02:01:02.188647 16897 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/data/info.pb
I20260929 02:01:02.191601 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 16897
I20260929 02:01:02.191802 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/instance
I20260929 02:01:02.197613 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.66:0
--local_ip_for_outbound_sockets=127.0.71.66
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:39181,127.0.71.125:36751
--builtin_ntp_servers=127.0.71.84:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 02:01:02.206470 17032 heartbeater.cc:344] Connected to a master server at 127.0.71.126:39181
I20260929 02:01:02.206498 17031 heartbeater.cc:344] Connected to a master server at 127.0.71.125:36751
I20260929 02:01:02.206892 17031 heartbeater.cc:461] Registering TS with master...
I20260929 02:01:02.206897 17032 heartbeater.cc:461] Registering TS with master...
I20260929 02:01:02.207911 17031 heartbeater.cc:507] Master 127.0.71.125:36751 requested a full tablet report, sending...
I20260929 02:01:02.208776 17032 heartbeater.cc:507] Master 127.0.71.126:39181 requested a full tablet report, sending...
I20260929 02:01:02.210413 16819 ts_manager.cc:194] Registered new tserver with Master: 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625)
I20260929 02:01:02.210448 16744 ts_manager.cc:194] Registered new tserver with Master: 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625)
I20260929 02:01:02.212164 16819 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.65:49525
I20260929 02:01:02.439848 16886 catalog_manager.cc:1308] Loaded cluster ID: 880bcdb43e214b76927cd2ef4af8291c
I20260929 02:01:02.439955 16886 catalog_manager.cc:1601] T 00000000000000000000000000000000 P 463404b2470c42c9b6bd67f027cf5e2b: loading cluster ID for follower catalog manager: success
I20260929 02:01:02.443542 16886 catalog_manager.cc:1623] T 00000000000000000000000000000000 P 463404b2470c42c9b6bd67f027cf5e2b: acquiring CA information for follower catalog manager: success
I20260929 02:01:02.446148 16886 catalog_manager.cc:1651] T 00000000000000000000000000000000 P 463404b2470c42c9b6bd67f027cf5e2b: importing token verification keys for follower catalog manager: success; most recent TSK sequence number 0
W20260929 02:01:02.543187 17038 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:01:02.543550 17038 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:01:02.543685 17038 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:01:02.552271 17038 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:01:02.552567 17038 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.66
I20260929 02:01:02.562626 17038 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.66:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
--webserver_interface=127.0.71.66
--webserver_port=0
--tserver_master_addrs=127.0.71.126:39181,127.0.71.125:36751
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.66
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:01:02.564188 17038 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:01:02.566056 17038 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:01:02.575207 17043 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:01:02.575241 17044 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:01:02.576047 17046 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:01:02.576740 17038 server_base.cc:1061] running on GCE node
I20260929 02:01:02.577275 17038 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:01:02.578192 17038 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:01:02.579432 17038 hybrid_clock.cc:648] HybridClock initialized: now 1790647262579351 us; error 52 us; skew 500 ppm
I20260929 02:01:02.579734 17038 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:01:02.581940 17038 webserver.cc:533] Webserver started at http://127.0.71.66:32903/ using document root <none> and password file <none>
I20260929 02:01:02.582657 17038 fs_manager.cc:362] Metadata directory not provided
I20260929 02:01:02.582787 17038 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:01:02.583151 17038 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:01:02.585500 17038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/data/instance:
uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
format_stamp: "Formatted at 2026-09-29 02:01:02 on dist-test-slave-vz23"
I20260929 02:01:02.586401 17038 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/instance:
uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
format_stamp: "Formatted at 2026-09-29 02:01:02 on dist-test-slave-vz23"
I20260929 02:01:02.591717 17038 fs_manager.cc:696] Time spent creating directory manager: real 0.005s user 0.005s sys 0.001s
I20260929 02:01:02.595017 17052 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:01:02.596659 17038 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:01:02.596853 17038 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal
uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
format_stamp: "Formatted at 2026-09-29 02:01:02 on dist-test-slave-vz23"
I20260929 02:01:02.597111 17038 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:01:02.612865 17038 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:01:02.613853 17038 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:01:02.614127 17038 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:01:02.615089 17038 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 02:01:02.616971 17038 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:01:02.617105 17038 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:01:02.617223 17038 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 02:01:02.617313 17038 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:01:02.656414 17038 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.66:42649
I20260929 02:01:02.656535 17164 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.66:42649 every 8 connection(s)
I20260929 02:01:02.657954 17038 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/data/info.pb
I20260929 02:01:02.668378 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 17038
I20260929 02:01:02.668604 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/instance
I20260929 02:01:02.673144 285 external_mini_cluster.cc:1372] Running /tmp/dist-test-taskniqJIA/build/asan/bin/kudu
/tmp/dist-test-taskniqJIA/build/asan/bin/kudu
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/data
--block_manager=log
--webserver_interface=localhost
--never_fsync
--enable_minidumps=false
--redact=none
--metrics_log_interval_ms=1000
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/logs
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--server_dump_info_format=pb
--rpc_server_allow_ephemeral_ports
--unlock_experimental_flags
--unlock_unsafe_flags
--logtostderr
--logbuflevel=-1
--ipki_server_key_size=768
--openssl_security_level_override=0
tserver
run
--rpc_bind_addresses=127.0.71.67:0
--local_ip_for_outbound_sockets=127.0.71.67
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:39181,127.0.71.125:36751
--builtin_ntp_servers=127.0.71.84:35907
--builtin_ntp_poll_interval_ms=100
--ntp_initial_sync_wait_secs=10
--time_source=builtin with env {}
I20260929 02:01:02.673184 17166 heartbeater.cc:344] Connected to a master server at 127.0.71.126:39181
I20260929 02:01:02.673584 17166 heartbeater.cc:461] Registering TS with master...
I20260929 02:01:02.674638 17166 heartbeater.cc:507] Master 127.0.71.126:39181 requested a full tablet report, sending...
I20260929 02:01:02.675552 17165 heartbeater.cc:344] Connected to a master server at 127.0.71.125:36751
I20260929 02:01:02.675684 17165 heartbeater.cc:461] Registering TS with master...
I20260929 02:01:02.676204 17165 heartbeater.cc:507] Master 127.0.71.125:36751 requested a full tablet report, sending...
I20260929 02:01:02.676662 16744 ts_manager.cc:194] Registered new tserver with Master: 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:02.677434 16819 ts_manager.cc:194] Registered new tserver with Master: 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:02.678290 16819 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.66:44433
W20260929 02:01:03.017743 17170 flags.cc:432] Enabled unsafe flag: --openssl_security_level_override=0
W20260929 02:01:03.018141 17170 flags.cc:432] Enabled unsafe flag: --rpc_server_allow_ephemeral_ports=true
W20260929 02:01:03.018286 17170 flags.cc:432] Enabled unsafe flag: --never_fsync=true
W20260929 02:01:03.027256 17170 flags.cc:432] Enabled experimental flag: --ipki_server_key_size=768
W20260929 02:01:03.027509 17170 flags.cc:432] Enabled experimental flag: --local_ip_for_outbound_sockets=127.0.71.67
I20260929 02:01:03.038622 17170 tablet_server_runner.cc:78] Tablet server non-default flags:
--log_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/logs
--logbuflevel=-1
--logtostderr=true
--builtin_ntp_poll_interval_ms=100
--builtin_ntp_servers=127.0.71.84:35907
--ntp_initial_sync_wait_secs=10
--time_source=builtin
--fs_data_dirs=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/data
--fs_wal_dir=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal
--ipki_server_key_size=768
--openssl_security_level_override=0
--rpc_bind_addresses=127.0.71.67:0
--rpc_server_allow_ephemeral_ports=true
--metrics_log_interval_ms=1000
--server_dump_info_format=pb
--server_dump_info_path=/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
--webserver_interface=127.0.71.67
--webserver_port=0
--tserver_master_addrs=127.0.71.126:39181,127.0.71.125:36751
--never_fsync=true
--redact=none
--unlock_experimental_flags=true
--unlock_unsafe_flags=true
--enable_minidumps=false
--local_ip_for_outbound_sockets=127.0.71.67
Tablet server version:
kudu 1.19.0-SNAPSHOT
revision 3abfc3e270b48c5a2e69ad8b9aebc64e2a8e2644
build type FASTDEBUG
built by None at 29 Sep 2026 01:43:23 UTC on bdcb31816ec0
build id 14049
ASAN enabled
I20260929 02:01:03.040306 17170 env_posix.cc:2285] Not raising this process' open files per process limit of 1048576; it is already as high as it can go
I20260929 02:01:03.042193 17170 file_cache.cc:504] Constructed file cache file cache with capacity 419430
W20260929 02:01:03.052814 17176 instance_detector.cc:116] could not retrieve Azure instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:01:03.053206 17175 instance_detector.cc:116] could not retrieve AWS instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
W20260929 02:01:03.053773 17178 instance_detector.cc:116] could not retrieve OpenStack instance metadata: Network error: curl error: HTTP response code said error: The requested URL returned error: 404
I20260929 02:01:03.054603 17170 server_base.cc:1061] running on GCE node
I20260929 02:01:03.055380 17170 hybrid_clock.cc:584] initializing the hybrid clock with 'builtin' time source
I20260929 02:01:03.056607 17170 hybrid_clock.cc:630] waiting up to --ntp_initial_sync_wait_secs=10 seconds for the clock to synchronize
I20260929 02:01:03.057858 17170 hybrid_clock.cc:648] HybridClock initialized: now 1790647263057793 us; error 43 us; skew 500 ppm
I20260929 02:01:03.058202 17170 server_base.cc:861] Flag tcmalloc_max_total_thread_cache_bytes is not working since tcmalloc is not enabled.
I20260929 02:01:03.060602 17170 webserver.cc:533] Webserver started at http://127.0.71.67:34309/ using document root <none> and password file <none>
I20260929 02:01:03.061393 17170 fs_manager.cc:362] Metadata directory not provided
I20260929 02:01:03.061494 17170 fs_manager.cc:368] Using write-ahead log directory (fs_wal_dir) as metadata directory
I20260929 02:01:03.061841 17170 server_base.cc:909] This appears to be a new deployment of Kudu; creating new FS layout
I20260929 02:01:03.064705 17170 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/data/instance:
uuid: "d103dd4823044faebe46100a0dfbee88"
format_stamp: "Formatted at 2026-09-29 02:01:03 on dist-test-slave-vz23"
I20260929 02:01:03.065577 17170 fs_manager.cc:1068] Generated new instance metadata in path /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/instance:
uuid: "d103dd4823044faebe46100a0dfbee88"
format_stamp: "Formatted at 2026-09-29 02:01:03 on dist-test-slave-vz23"
I20260929 02:01:03.072019 17170 fs_manager.cc:696] Time spent creating directory manager: real 0.006s user 0.006s sys 0.000s
I20260929 02:01:03.075820 17185 log_block_manager.cc:4127] Time spent loading block containers with low live blocks: real 0.000s user 0.000s sys 0.000s
I20260929 02:01:03.077349 17170 fs_manager.cc:730] Time spent opening block manager: real 0.003s user 0.003s sys 0.000s
I20260929 02:01:03.077661 17170 fs_manager.cc:647] Opened local filesystem: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/data,/tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal
uuid: "d103dd4823044faebe46100a0dfbee88"
format_stamp: "Formatted at 2026-09-29 02:01:03 on dist-test-slave-vz23"
I20260929 02:01:03.077943 17170 fs_report.cc:389] FS layout report
--------------------
wal directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal
metadata directory: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal
1 data directories: /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/data/data
Total live blocks: 0
Total live bytes: 0
Total live bytes (after alignment): 0
Total number of LBM containers: 0 (0 full)
Did not check for missing blocks
Did not check for orphaned blocks
Total full LBM containers with extra space: 0 (0 repaired)
Total full LBM container extra space in bytes: 0 (0 repaired)
Total incomplete LBM containers: 0 (0 repaired)
Total LBM partial records: 0 (0 repaired)
Total corrupted LBM metadata records in RocksDB: 0 (0 repaired)
I20260929 02:01:03.102660 17170 rpc_server.cc:225] running with OpenSSL 1.1.1 11 Sep 2018
I20260929 02:01:03.103570 17170 env_posix.cc:2285] Not raising this process' running threads per effective uid limit of 18446744073709551615; it is already as high as it can go
I20260929 02:01:03.103858 17170 kserver.cc:163] Server-wide thread pool size limit: 3276
I20260929 02:01:03.105027 17170 txn_system_client.cc:432] TxnSystemClient initialization is disabled...
I20260929 02:01:03.106702 17170 ts_tablet_manager.cc:631] Loaded tablet metadata (0 total tablets, 0 live tablets)
I20260929 02:01:03.106832 17170 ts_tablet_manager.cc:577] Time spent load tablet metadata: real 0.000s user 0.000s sys 0.000s
I20260929 02:01:03.106966 17170 ts_tablet_manager.cc:662] Registered 0 tablets
I20260929 02:01:03.107060 17170 ts_tablet_manager.cc:641] Time spent register tablets: real 0.000s user 0.000s sys 0.000s
I20260929 02:01:03.143416 17170 rpc_server.cc:307] RPC server started. Bound to: 127.0.71.67:44233
I20260929 02:01:03.143505 17297 acceptor_pool.cc:272] collecting diagnostics on the listening RPC socket 127.0.71.67:44233 every 8 connection(s)
I20260929 02:01:03.144953 17170 server_base.cc:1193] Dumped server information to /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/data/info.pb
I20260929 02:01:03.153002 285 external_mini_cluster.cc:1434] Started /tmp/dist-test-taskniqJIA/build/asan/bin/kudu as pid 17170
I20260929 02:01:03.153211 285 external_mini_cluster.cc:1448] Reading /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/instance
I20260929 02:01:03.161859 17298 heartbeater.cc:344] Connected to a master server at 127.0.71.125:36751
I20260929 02:01:03.162221 17298 heartbeater.cc:461] Registering TS with master...
I20260929 02:01:03.163039 17298 heartbeater.cc:507] Master 127.0.71.125:36751 requested a full tablet report, sending...
I20260929 02:01:03.164008 17299 heartbeater.cc:344] Connected to a master server at 127.0.71.126:39181
I20260929 02:01:03.164151 17299 heartbeater.cc:461] Registering TS with master...
I20260929 02:01:03.164662 17299 heartbeater.cc:507] Master 127.0.71.126:39181 requested a full tablet report, sending...
I20260929 02:01:03.164664 16819 ts_manager.cc:194] Registered new tserver with Master: d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:03.165663 16744 ts_manager.cc:194] Registered new tserver with Master: d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:03.165724 16819 master_service.cc:502] Signed X509 certificate for tserver {username='slave'} at 127.0.71.67:47643
I20260929 02:01:03.174962 285 external_mini_cluster.cc:955] 3 TS(s) registered with all masters
I20260929 02:01:03.198539 16819 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44708:
name: "table-0"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "col"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 16
seed: 0
}
}
W20260929 02:01:03.200197 16819 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table-0 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:01:03.215170 17031 heartbeater.cc:499] Master 127.0.71.125:36751 was elected leader, sending a full tablet report...
I20260929 02:01:03.218107 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:03.218663 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:03.241693 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.023s user 0.013s sys 0.000s Metrics: {"bytes_written":6939,"cfile_init":1,"dirs.queue_time_us":469,"dirs.run_cpu_time_us":181,"dirs.run_wall_time_us":944,"drs_written":1,"lbm_read_time_us":97,"lbm_reads_lt_1ms":4,"lbm_write_time_us":647,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":17,"thread_start_us":295,"threads_started":1}
I20260929 02:01:03.243626 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:03.243999 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260929 02:01:03.244325 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:03.247630 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4}
I20260929 02:01:03.248072 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:03.249037 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.014994
I20260929 02:01:03.249686 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.014994
I20260929 02:01:03.250311 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.014994
I20260929 02:01:03.259981 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.015s user 0.002s sys 0.001s Metrics: {"bytes_written":697,"delete_count":0,"lbm_write_time_us":193,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":16}
I20260929 02:01:03.277861 16953 tablet_service.cc:1511] Processing CreateTablet for tablet 4d9e051c702b42bc9392c5dcae42265d (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.278968 16955 tablet_service.cc:1511] Processing CreateTablet for tablet 0aa899b966dd481088d18119ee57d81a (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.279517 16953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d9e051c702b42bc9392c5dcae42265d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.279742 16956 tablet_service.cc:1511] Processing CreateTablet for tablet c64e85b42e0d4618885975ac4125d4f2 (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.280190 16956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c64e85b42e0d4618885975ac4125d4f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.278236 16954 tablet_service.cc:1511] Processing CreateTablet for tablet 43f983a2802b4ad3871d743b42b5deea (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.280843 16954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43f983a2802b4ad3871d743b42b5deea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.282003 16957 tablet_service.cc:1511] Processing CreateTablet for tablet c7189d73aa7c43b99293f2da0e47338b (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.282505 16957 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7189d73aa7c43b99293f2da0e47338b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.283656 16958 tablet_service.cc:1511] Processing CreateTablet for tablet 3cb85eb3aa494f21bc8e506004318650 (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.284189 16958 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cb85eb3aa494f21bc8e506004318650. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.288195 16805 tablet.cc:2133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260929 02:01:03.288940 16959 tablet_service.cc:1511] Processing CreateTablet for tablet 8601b0ded51b49d0abeb95f7e5fedf8c (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.289413 16959 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8601b0ded51b49d0abeb95f7e5fedf8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.292493 16966 tablet_service.cc:1511] Processing CreateTablet for tablet c3686e82fe0f4b6790a9a7f290c06876 (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.292979 16960 tablet_service.cc:1511] Processing CreateTablet for tablet b591026c764b48a9af227b7be6dd0ee6 (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.293742 16961 tablet_service.cc:1511] Processing CreateTablet for tablet b2d556cada174f83a633a7a8836826ce (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.294516 16962 tablet_service.cc:1511] Processing CreateTablet for tablet 7a962f60729242a1bcebe88c89f8f895 (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.292994 16966 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3686e82fe0f4b6790a9a7f290c06876. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.295266 16963 tablet_service.cc:1511] Processing CreateTablet for tablet ca4fa192c95745bfa163ef63f98d29a5 (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.295660 16963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca4fa192c95745bfa163ef63f98d29a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.301801 16964 tablet_service.cc:1511] Processing CreateTablet for tablet 9a0e71f5f586410fabe38e3c3b26dfd0 (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.302222 16964 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a0e71f5f586410fabe38e3c3b26dfd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.305926 16965 tablet_service.cc:1511] Processing CreateTablet for tablet 9103285f38f149a8ab80fdd0eb792dee (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.306401 16961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2d556cada174f83a633a7a8836826ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.307902 16965 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9103285f38f149a8ab80fdd0eb792dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.311726 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.045s user 0.010s sys 0.000s Metrics: {"dirs.queue_time_us":11038,"dirs.run_cpu_time_us":759,"dirs.run_wall_time_us":46108,"lbm_write_time_us":372,"lbm_writes_lt_1ms":7,"num_input_rowsets":0,"thread_start_us":5189,"threads_started":1}
I20260929 02:01:03.311914 17233 tablet_service.cc:1511] Processing CreateTablet for tablet c3686e82fe0f4b6790a9a7f290c06876 (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.313458 16962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a962f60729242a1bcebe88c89f8f895. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.313380 16952 tablet_service.cc:1511] Processing CreateTablet for tablet 2bc3dbd1c82e4d2cb9577c51d9b19224 (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.313887 17233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3686e82fe0f4b6790a9a7f290c06876. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.316627 16955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0aa899b966dd481088d18119ee57d81a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.316517 17232 tablet_service.cc:1511] Processing CreateTablet for tablet 9103285f38f149a8ab80fdd0eb792dee (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.316967 16952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bc3dbd1c82e4d2cb9577c51d9b19224. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.317124 17232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9103285f38f149a8ab80fdd0eb792dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.317255 17224 tablet_service.cc:1511] Processing CreateTablet for tablet c7189d73aa7c43b99293f2da0e47338b (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.317530 16951 tablet_service.cc:1511] Processing CreateTablet for tablet 37e07f15e43c49a4bf5ab1334dd85d22 (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.317728 17224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7189d73aa7c43b99293f2da0e47338b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.317972 16951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37e07f15e43c49a4bf5ab1334dd85d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.318050 17231 tablet_service.cc:1511] Processing CreateTablet for tablet 9a0e71f5f586410fabe38e3c3b26dfd0 (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.318559 17231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a0e71f5f586410fabe38e3c3b26dfd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.318524 17230 tablet_service.cc:1511] Processing CreateTablet for tablet ca4fa192c95745bfa163ef63f98d29a5 (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.319046 17230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca4fa192c95745bfa163ef63f98d29a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.319546 17229 tablet_service.cc:1511] Processing CreateTablet for tablet 7a962f60729242a1bcebe88c89f8f895 (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.320014 17229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a962f60729242a1bcebe88c89f8f895. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.320513 17226 tablet_service.cc:1511] Processing CreateTablet for tablet 8601b0ded51b49d0abeb95f7e5fedf8c (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.321004 17226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8601b0ded51b49d0abeb95f7e5fedf8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.320968 17222 tablet_service.cc:1511] Processing CreateTablet for tablet 0aa899b966dd481088d18119ee57d81a (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.322649 16960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b591026c764b48a9af227b7be6dd0ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.320011 17228 tablet_service.cc:1511] Processing CreateTablet for tablet b2d556cada174f83a633a7a8836826ce (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.323164 17228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2d556cada174f83a633a7a8836826ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.327798 17330 tablet_bootstrap.cc:492] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:03.331218 17330 tablet_bootstrap.cc:654] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.332702 17227 tablet_service.cc:1511] Processing CreateTablet for tablet b591026c764b48a9af227b7be6dd0ee6 (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.333818 17225 tablet_service.cc:1511] Processing CreateTablet for tablet 3cb85eb3aa494f21bc8e506004318650 (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.333943 17222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0aa899b966dd481088d18119ee57d81a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.336259 17220 tablet_service.cc:1511] Processing CreateTablet for tablet 4d9e051c702b42bc9392c5dcae42265d (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.336746 17219 tablet_service.cc:1511] Processing CreateTablet for tablet 2bc3dbd1c82e4d2cb9577c51d9b19224 (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.337772 17223 tablet_service.cc:1511] Processing CreateTablet for tablet c64e85b42e0d4618885975ac4125d4f2 (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.339136 17221 tablet_service.cc:1511] Processing CreateTablet for tablet 43f983a2802b4ad3871d743b42b5deea (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.339682 17218 tablet_service.cc:1511] Processing CreateTablet for tablet 37e07f15e43c49a4bf5ab1334dd85d22 (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.340161 17220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d9e051c702b42bc9392c5dcae42265d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.342559 17227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b591026c764b48a9af227b7be6dd0ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.345217 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.124s user 0.018s sys 0.007s Metrics: {"bytes_written":697,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":63211,"dirs.run_cpu_time_us":152,"dirs.run_wall_time_us":7121,"drs_written":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4,"lbm_write_time_us":898,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"reinsert_count":0,"rows_written":17,"thread_start_us":63342,"threads_started":1,"update_count":16}
I20260929 02:01:03.346446 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260929 02:01:03.346812 17221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43f983a2802b4ad3871d743b42b5deea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.346776 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.349724
I20260929 02:01:03.347121 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.349724
I20260929 02:01:03.347424 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260929 02:01:03.347710 17219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bc3dbd1c82e4d2cb9577c51d9b19224. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.348166 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":134,"lbm_reads_lt_1ms":4}
I20260929 02:01:03.352610 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:03.352972 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:03.353984 17218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37e07f15e43c49a4bf5ab1334dd85d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.359938 17092 tablet_service.cc:1511] Processing CreateTablet for tablet 3cb85eb3aa494f21bc8e506004318650 (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.360847 17088 tablet_service.cc:1511] Processing CreateTablet for tablet 43f983a2802b4ad3871d743b42b5deea (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.361545 17092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cb85eb3aa494f21bc8e506004318650. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.361604 17087 tablet_service.cc:1511] Processing CreateTablet for tablet 4d9e051c702b42bc9392c5dcae42265d (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.362051 17087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4d9e051c702b42bc9392c5dcae42265d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.363410 17097 tablet_service.cc:1511] Processing CreateTablet for tablet ca4fa192c95745bfa163ef63f98d29a5 (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.363966 17097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ca4fa192c95745bfa163ef63f98d29a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.364924 17086 tablet_service.cc:1511] Processing CreateTablet for tablet 2bc3dbd1c82e4d2cb9577c51d9b19224 (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.365262 17093 tablet_service.cc:1511] Processing CreateTablet for tablet 8601b0ded51b49d0abeb95f7e5fedf8c (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.365298 17225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3cb85eb3aa494f21bc8e506004318650. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.365381 17086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2bc3dbd1c82e4d2cb9577c51d9b19224. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.365762 17093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8601b0ded51b49d0abeb95f7e5fedf8c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.366171 17085 tablet_service.cc:1511] Processing CreateTablet for tablet 37e07f15e43c49a4bf5ab1334dd85d22 (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.366641 17085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 37e07f15e43c49a4bf5ab1334dd85d22. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.366777 17091 tablet_service.cc:1511] Processing CreateTablet for tablet c7189d73aa7c43b99293f2da0e47338b (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.367161 17091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7189d73aa7c43b99293f2da0e47338b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.367745 17099 tablet_service.cc:1511] Processing CreateTablet for tablet 9103285f38f149a8ab80fdd0eb792dee (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.368175 17099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9103285f38f149a8ab80fdd0eb792dee. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.368518 17095 tablet_service.cc:1511] Processing CreateTablet for tablet b2d556cada174f83a633a7a8836826ce (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.368856 17094 tablet_service.cc:1511] Processing CreateTablet for tablet b591026c764b48a9af227b7be6dd0ee6 (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.368973 17095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b2d556cada174f83a633a7a8836826ce. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.369304 17094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b591026c764b48a9af227b7be6dd0ee6. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.369303 17223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c64e85b42e0d4618885975ac4125d4f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.368363 17100 tablet_service.cc:1511] Processing CreateTablet for tablet c3686e82fe0f4b6790a9a7f290c06876 (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.369840 17100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c3686e82fe0f4b6790a9a7f290c06876. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.368523 17330 log.cc:826] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f: Log is configured to *not* fsync() on all Append() calls
I20260929 02:01:03.369966 17096 tablet_service.cc:1511] Processing CreateTablet for tablet 7a962f60729242a1bcebe88c89f8f895 (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.368084 17098 tablet_service.cc:1511] Processing CreateTablet for tablet 9a0e71f5f586410fabe38e3c3b26dfd0 (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.370381 17096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7a962f60729242a1bcebe88c89f8f895. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.370664 17098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9a0e71f5f586410fabe38e3c3b26dfd0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.360312 17089 tablet_service.cc:1511] Processing CreateTablet for tablet 0aa899b966dd481088d18119ee57d81a (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.371464 17089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 0aa899b966dd481088d18119ee57d81a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.373672 17088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 43f983a2802b4ad3871d743b42b5deea. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.374081 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 640 bytes on disk
I20260929 02:01:03.374485 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 640 bytes on disk
I20260929 02:01:03.375430 17090 tablet_service.cc:1511] Processing CreateTablet for tablet c64e85b42e0d4618885975ac4125d4f2 (DEFAULT_TABLE table=table-0 [id=b262d8c39bfe4f0abba8b155fe000610]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:03.375535 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":95,"lbm_reads_lt_1ms":4}
I20260929 02:01:03.375886 17090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c64e85b42e0d4618885975ac4125d4f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:03.375988 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:03.432902 17334 tablet_bootstrap.cc:492] T b2d556cada174f83a633a7a8836826ce P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:03.447307 17335 tablet_bootstrap.cc:492] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:03.449407 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.101s user 0.016s sys 0.012s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":1677,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":3342,"dirs.run_cpu_time_us":289,"dirs.run_wall_time_us":3453,"lbm_read_time_us":502,"lbm_reads_lt_1ms":21,"lbm_write_time_us":402,"lbm_writes_lt_1ms":9,"peak_mem_usage":32768,"reinsert_count":0,"thread_start_us":77658,"threads_started":1,"update_count":16}
I20260929 02:01:03.451485 17335 tablet_bootstrap.cc:654] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.451848 16804 tablet.cc:2133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260929 02:01:03.452890 17335 log.cc:826] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 8668afa5b5ca4ebdabc812d0789de7bc: Log is configured to *not* fsync() on all Append() calls
I20260929 02:01:03.453431 17330 tablet_bootstrap.cc:492] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:03.453860 17330 ts_tablet_manager.cc:1449] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.126s user 0.005s sys 0.017s
I20260929 02:01:03.454129 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.204s user 0.024s sys 0.017s Metrics: {"bytes_written":11879,"cfile_cache_hit":12,"cfile_cache_hit_bytes":3106,"cfile_cache_miss":30,"cfile_cache_miss_bytes":4618,"cfile_init":8,"delta_iterators_relevant":5,"dirs.queue_time_us":27821,"dirs.run_cpu_time_us":1079,"dirs.run_wall_time_us":12188,"drs_written":1,"lbm_read_time_us":16791,"lbm_reads_1-10_ms":1,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":60,"lbm_write_time_us":1033,"lbm_writes_lt_1ms":27,"mutex_wait_us":17062,"num_input_rowsets":2,"peak_mem_usage":65536,"rows_written":20,"thread_start_us":78548,"threads_started":1}
I20260929 02:01:03.457808 17335 tablet_bootstrap.cc:492] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:03.458073 17335 ts_tablet_manager.cc:1449] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.011s user 0.009s sys 0.000s
I20260929 02:01:03.459090 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.207s user 0.017s sys 0.025s Metrics: {"dirs.queue_time_us":26472,"dirs.run_cpu_time_us":988,"dirs.run_wall_time_us":6874,"lbm_write_time_us":354,"lbm_writes_lt_1ms":7,"mutex_wait_us":5177,"num_input_rowsets":0,"thread_start_us":174212,"threads_started":3}
I20260929 02:01:03.462765 17334 tablet_bootstrap.cc:654] T b2d556cada174f83a633a7a8836826ce P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.463081 17330 raft_consensus.cc:359] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.463439 17330 raft_consensus.cc:385] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.463549 17330 raft_consensus.cc:740] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.464242 17330 consensus_queue.cc:260] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.465464 17335 raft_consensus.cc:359] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.465860 17335 raft_consensus.cc:385] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.465955 17335 raft_consensus.cc:740] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.466748 17335 consensus_queue.cc:260] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.471989 17334 log.cc:826] T b2d556cada174f83a633a7a8836826ce P d103dd4823044faebe46100a0dfbee88: Log is configured to *not* fsync() on all Append() calls
I20260929 02:01:03.472826 17330 ts_tablet_manager.cc:1480] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.019s user 0.012s sys 0.007s
I20260929 02:01:03.474014 17165 heartbeater.cc:499] Master 127.0.71.125:36751 was elected leader, sending a full tablet report...
I20260929 02:01:03.475668 17330 tablet_bootstrap.cc:492] T c3686e82fe0f4b6790a9a7f290c06876 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:03.474768 17335 ts_tablet_manager.cc:1480] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.017s user 0.010s sys 0.007s
I20260929 02:01:03.477792 17335 tablet_bootstrap.cc:492] T 9103285f38f149a8ab80fdd0eb792dee P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:03.478370 17330 tablet_bootstrap.cc:654] T c3686e82fe0f4b6790a9a7f290c06876 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.479665 17335 tablet_bootstrap.cc:654] T 9103285f38f149a8ab80fdd0eb792dee P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.485800 17334 tablet_bootstrap.cc:492] T b2d556cada174f83a633a7a8836826ce P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:03.486153 17334 ts_tablet_manager.cc:1449] T b2d556cada174f83a633a7a8836826ce P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.007s
I20260929 02:01:03.491957 17330 tablet_bootstrap.cc:492] T c3686e82fe0f4b6790a9a7f290c06876 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:03.492143 17330 ts_tablet_manager.cc:1449] T c3686e82fe0f4b6790a9a7f290c06876 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.010s
I20260929 02:01:03.493137 17335 tablet_bootstrap.cc:492] T 9103285f38f149a8ab80fdd0eb792dee P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:03.493134 17330 raft_consensus.cc:359] T c3686e82fe0f4b6790a9a7f290c06876 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } }
I20260929 02:01:03.493310 17335 ts_tablet_manager.cc:1449] T 9103285f38f149a8ab80fdd0eb792dee P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.009s
I20260929 02:01:03.493351 17330 raft_consensus.cc:385] T c3686e82fe0f4b6790a9a7f290c06876 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.493459 17330 raft_consensus.cc:740] T c3686e82fe0f4b6790a9a7f290c06876 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.493446 17334 raft_consensus.cc:359] T b2d556cada174f83a633a7a8836826ce P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.493738 17330 consensus_queue.cc:260] T c3686e82fe0f4b6790a9a7f290c06876 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } }
I20260929 02:01:03.493858 17334 raft_consensus.cc:385] T b2d556cada174f83a633a7a8836826ce P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.494261 17330 ts_tablet_manager.cc:1480] T c3686e82fe0f4b6790a9a7f290c06876 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 02:01:03.494374 17335 raft_consensus.cc:359] T 9103285f38f149a8ab80fdd0eb792dee P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.494546 17335 raft_consensus.cc:385] T 9103285f38f149a8ab80fdd0eb792dee P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.494623 17330 tablet_bootstrap.cc:492] T 3cb85eb3aa494f21bc8e506004318650 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:03.494632 17335 raft_consensus.cc:740] T 9103285f38f149a8ab80fdd0eb792dee P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.494957 17335 consensus_queue.cc:260] T 9103285f38f149a8ab80fdd0eb792dee P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.495590 17335 ts_tablet_manager.cc:1480] T 9103285f38f149a8ab80fdd0eb792dee P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 02:01:03.495908 17335 tablet_bootstrap.cc:492] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:03.496552 17330 tablet_bootstrap.cc:654] T 3cb85eb3aa494f21bc8e506004318650 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.497803 17335 tablet_bootstrap.cc:654] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.493973 17334 raft_consensus.cc:740] T b2d556cada174f83a633a7a8836826ce P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.501353 17334 consensus_queue.cc:260] T b2d556cada174f83a633a7a8836826ce P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.501998 17335 tablet_bootstrap.cc:492] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:03.502135 17330 tablet_bootstrap.cc:492] T 3cb85eb3aa494f21bc8e506004318650 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:03.502167 17335 ts_tablet_manager.cc:1449] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.006s user 0.002s sys 0.003s
I20260929 02:01:03.502297 17330 ts_tablet_manager.cc:1449] T 3cb85eb3aa494f21bc8e506004318650 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.008s user 0.000s sys 0.006s
I20260929 02:01:03.502720 17298 heartbeater.cc:499] Master 127.0.71.125:36751 was elected leader, sending a full tablet report...
I20260929 02:01:03.502867 17335 raft_consensus.cc:359] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.503062 17335 raft_consensus.cc:385] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.503140 17335 raft_consensus.cc:740] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.503059 17330 raft_consensus.cc:359] T 3cb85eb3aa494f21bc8e506004318650 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.503222 17334 ts_tablet_manager.cc:1480] T b2d556cada174f83a633a7a8836826ce P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.017s user 0.000s sys 0.009s
I20260929 02:01:03.503239 17330 raft_consensus.cc:385] T 3cb85eb3aa494f21bc8e506004318650 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.503322 17330 raft_consensus.cc:740] T 3cb85eb3aa494f21bc8e506004318650 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.503373 17335 consensus_queue.cc:260] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.503615 17330 consensus_queue.cc:260] T 3cb85eb3aa494f21bc8e506004318650 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.503959 17335 ts_tablet_manager.cc:1480] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 02:01:03.504184 17330 ts_tablet_manager.cc:1480] T 3cb85eb3aa494f21bc8e506004318650 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 02:01:03.504258 17335 tablet_bootstrap.cc:492] T c7189d73aa7c43b99293f2da0e47338b P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:03.504489 17330 tablet_bootstrap.cc:492] T 8601b0ded51b49d0abeb95f7e5fedf8c P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:03.506168 17335 tablet_bootstrap.cc:654] T c7189d73aa7c43b99293f2da0e47338b P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.506285 17330 tablet_bootstrap.cc:654] T 8601b0ded51b49d0abeb95f7e5fedf8c P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.509232 17334 tablet_bootstrap.cc:492] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:03.511198 17334 tablet_bootstrap.cc:654] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.514982 17334 tablet_bootstrap.cc:492] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:03.515144 17334 ts_tablet_manager.cc:1449] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.006s user 0.000s sys 0.004s
I20260929 02:01:03.516044 17334 raft_consensus.cc:359] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.516216 17334 raft_consensus.cc:385] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.516301 17334 raft_consensus.cc:740] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.516713 17334 consensus_queue.cc:260] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.518297 17335 tablet_bootstrap.cc:492] T c7189d73aa7c43b99293f2da0e47338b P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:03.518476 17335 ts_tablet_manager.cc:1449] T c7189d73aa7c43b99293f2da0e47338b P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.014s user 0.009s sys 0.002s
I20260929 02:01:03.520056 17335 raft_consensus.cc:359] T c7189d73aa7c43b99293f2da0e47338b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.520277 17335 raft_consensus.cc:385] T c7189d73aa7c43b99293f2da0e47338b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.520398 17335 raft_consensus.cc:740] T c7189d73aa7c43b99293f2da0e47338b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.520710 17335 consensus_queue.cc:260] T c7189d73aa7c43b99293f2da0e47338b P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.520857 17330 tablet_bootstrap.cc:492] T 8601b0ded51b49d0abeb95f7e5fedf8c P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:03.521001 17330 ts_tablet_manager.cc:1449] T 8601b0ded51b49d0abeb95f7e5fedf8c P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.017s user 0.005s sys 0.009s
I20260929 02:01:03.521286 17335 ts_tablet_manager.cc:1480] T c7189d73aa7c43b99293f2da0e47338b P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:01:03.521560 17335 tablet_bootstrap.cc:492] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:03.521785 17330 raft_consensus.cc:359] T 8601b0ded51b49d0abeb95f7e5fedf8c P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.521960 17330 raft_consensus.cc:385] T 8601b0ded51b49d0abeb95f7e5fedf8c P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.522061 17330 raft_consensus.cc:740] T 8601b0ded51b49d0abeb95f7e5fedf8c P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.522284 17330 consensus_queue.cc:260] T 8601b0ded51b49d0abeb95f7e5fedf8c P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.522506 17334 ts_tablet_manager.cc:1480] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.007s user 0.003s sys 0.000s
I20260929 02:01:03.522750 17334 tablet_bootstrap.cc:492] T c7189d73aa7c43b99293f2da0e47338b P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:03.522913 17330 ts_tablet_manager.cc:1480] T 8601b0ded51b49d0abeb95f7e5fedf8c P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:01:03.523170 17330 tablet_bootstrap.cc:492] T 0aa899b966dd481088d18119ee57d81a P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:03.524531 17334 tablet_bootstrap.cc:654] T c7189d73aa7c43b99293f2da0e47338b P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.525835 17330 tablet_bootstrap.cc:654] T 0aa899b966dd481088d18119ee57d81a P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.527077 17335 tablet_bootstrap.cc:654] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.529958 17338 raft_consensus.cc:493] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:03.530242 17338 raft_consensus.cc:515] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.532591 17338 leader_election.cc:290] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:03.533555 17334 tablet_bootstrap.cc:492] T c7189d73aa7c43b99293f2da0e47338b P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:03.533612 17335 tablet_bootstrap.cc:492] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:03.533732 17334 ts_tablet_manager.cc:1449] T c7189d73aa7c43b99293f2da0e47338b P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.011s user 0.000s sys 0.004s
I20260929 02:01:03.533771 17330 tablet_bootstrap.cc:492] T 0aa899b966dd481088d18119ee57d81a P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:03.533804 17335 ts_tablet_manager.cc:1449] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.012s user 0.005s sys 0.001s
I20260929 02:01:03.533942 17330 ts_tablet_manager.cc:1449] T 0aa899b966dd481088d18119ee57d81a P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20260929 02:01:03.534600 17334 raft_consensus.cc:359] T c7189d73aa7c43b99293f2da0e47338b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.534600 17335 raft_consensus.cc:359] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.534677 17330 raft_consensus.cc:359] T 0aa899b966dd481088d18119ee57d81a P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.534854 17334 raft_consensus.cc:385] T c7189d73aa7c43b99293f2da0e47338b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.534857 17335 raft_consensus.cc:385] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.534904 17330 raft_consensus.cc:385] T 0aa899b966dd481088d18119ee57d81a P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.534932 17334 raft_consensus.cc:740] T c7189d73aa7c43b99293f2da0e47338b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.534947 17335 raft_consensus.cc:740] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.535136 17330 raft_consensus.cc:740] T 0aa899b966dd481088d18119ee57d81a P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.535189 17335 consensus_queue.cc:260] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.535230 17334 consensus_queue.cc:260] T c7189d73aa7c43b99293f2da0e47338b P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.535393 17330 consensus_queue.cc:260] T 0aa899b966dd481088d18119ee57d81a P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.535737 17335 ts_tablet_manager.cc:1480] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 02:01:03.536051 17335 tablet_bootstrap.cc:492] T b591026c764b48a9af227b7be6dd0ee6 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:03.537930 17335 tablet_bootstrap.cc:654] T b591026c764b48a9af227b7be6dd0ee6 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.537962 17330 ts_tablet_manager.cc:1480] T 0aa899b966dd481088d18119ee57d81a P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260929 02:01:03.538875 17334 ts_tablet_manager.cc:1480] T c7189d73aa7c43b99293f2da0e47338b P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.005s user 0.001s sys 0.002s
I20260929 02:01:03.539544 17330 tablet_bootstrap.cc:492] T ca4fa192c95745bfa163ef63f98d29a5 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:03.540023 17334 tablet_bootstrap.cc:492] T 7a962f60729242a1bcebe88c89f8f895 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:03.541884 17334 tablet_bootstrap.cc:654] T 7a962f60729242a1bcebe88c89f8f895 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.541985 17330 tablet_bootstrap.cc:654] T ca4fa192c95745bfa163ef63f98d29a5 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.544561 17334 tablet_bootstrap.cc:492] T 7a962f60729242a1bcebe88c89f8f895 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:03.544713 17334 ts_tablet_manager.cc:1449] T 7a962f60729242a1bcebe88c89f8f895 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.005s user 0.004s sys 0.000s
I20260929 02:01:03.545495 17334 raft_consensus.cc:359] T 7a962f60729242a1bcebe88c89f8f895 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.545706 17334 raft_consensus.cc:385] T 7a962f60729242a1bcebe88c89f8f895 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.545807 17334 raft_consensus.cc:740] T 7a962f60729242a1bcebe88c89f8f895 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.546108 17334 consensus_queue.cc:260] T 7a962f60729242a1bcebe88c89f8f895 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.548298 17334 ts_tablet_manager.cc:1480] T 7a962f60729242a1bcebe88c89f8f895 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:01:03.550222 17334 tablet_bootstrap.cc:492] T 9a0e71f5f586410fabe38e3c3b26dfd0 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:03.552409 17334 tablet_bootstrap.cc:654] T 9a0e71f5f586410fabe38e3c3b26dfd0 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.554983 17334 tablet_bootstrap.cc:492] T 9a0e71f5f586410fabe38e3c3b26dfd0 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:03.555169 17334 ts_tablet_manager.cc:1449] T 9a0e71f5f586410fabe38e3c3b26dfd0 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 02:01:03.555933 17334 raft_consensus.cc:359] T 9a0e71f5f586410fabe38e3c3b26dfd0 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.556130 17334 raft_consensus.cc:385] T 9a0e71f5f586410fabe38e3c3b26dfd0 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.556214 17334 raft_consensus.cc:740] T 9a0e71f5f586410fabe38e3c3b26dfd0 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.556484 17334 consensus_queue.cc:260] T 9a0e71f5f586410fabe38e3c3b26dfd0 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.557222 17330 tablet_bootstrap.cc:492] T ca4fa192c95745bfa163ef63f98d29a5 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:03.557705 17334 ts_tablet_manager.cc:1480] T 9a0e71f5f586410fabe38e3c3b26dfd0 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 02:01:03.558037 17334 tablet_bootstrap.cc:492] T 43f983a2802b4ad3871d743b42b5deea P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:03.559839 17334 tablet_bootstrap.cc:654] T 43f983a2802b4ad3871d743b42b5deea P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.560181 17330 ts_tablet_manager.cc:1449] T ca4fa192c95745bfa163ef63f98d29a5 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.021s user 0.009s sys 0.005s
I20260929 02:01:03.561237 17330 raft_consensus.cc:359] T ca4fa192c95745bfa163ef63f98d29a5 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.561563 17330 raft_consensus.cc:385] T ca4fa192c95745bfa163ef63f98d29a5 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.561723 17330 raft_consensus.cc:740] T ca4fa192c95745bfa163ef63f98d29a5 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.562259 17337 raft_consensus.cc:493] T 3cb85eb3aa494f21bc8e506004318650 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:03.562556 17330 consensus_queue.cc:260] T ca4fa192c95745bfa163ef63f98d29a5 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.562584 17337 raft_consensus.cc:515] T 3cb85eb3aa494f21bc8e506004318650 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.568005 17337 leader_election.cc:290] T 3cb85eb3aa494f21bc8e506004318650 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:03.644969 17338 raft_consensus.cc:493] T 9103285f38f149a8ab80fdd0eb792dee P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
W20260929 02:01:03.647719 17300 tablet.cc:2433] T 7a962f60729242a1bcebe88c89f8f895 P d103dd4823044faebe46100a0dfbee88: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:01:03.663182 17338 raft_consensus.cc:515] T 9103285f38f149a8ab80fdd0eb792dee P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
W20260929 02:01:03.668960 17167 tablet.cc:2433] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 8668afa5b5ca4ebdabc812d0789de7bc: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:01:03.670931 17338 leader_election.cc:290] T 9103285f38f149a8ab80fdd0eb792dee P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
W20260929 02:01:03.691550 17033 tablet.cc:2433] T 0aa899b966dd481088d18119ee57d81a P 97df55a0361841319adfae27c9787b2f: Can't schedule compaction. Clean time has not been advanced past its initial value.
I20260929 02:01:03.697258 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a0e71f5f586410fabe38e3c3b26dfd0" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:03.698352 16985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9103285f38f149a8ab80fdd0eb792dee" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:03.699059 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cb85eb3aa494f21bc8e506004318650" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
W20260929 02:01:03.699414 17056 leader_election.cc:343] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
W20260929 02:01:03.699797 17056 leader_election.cc:343] T 9103285f38f149a8ab80fdd0eb792dee P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
W20260929 02:01:03.700367 16921 leader_election.cc:343] T 3cb85eb3aa494f21bc8e506004318650 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:03.703271 17337 raft_consensus.cc:493] T 0aa899b966dd481088d18119ee57d81a P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:03.703524 17337 raft_consensus.cc:515] T 0aa899b966dd481088d18119ee57d81a P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.704457 17337 leader_election.cc:290] T 0aa899b966dd481088d18119ee57d81a P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:03.705397 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0aa899b966dd481088d18119ee57d81a" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
W20260929 02:01:03.706108 16921 leader_election.cc:343] T 0aa899b966dd481088d18119ee57d81a P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:03.710376 17120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cb85eb3aa494f21bc8e506004318650" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:03.710469 17119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0aa899b966dd481088d18119ee57d81a" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
W20260929 02:01:03.711849 16919 leader_election.cc:343] T 3cb85eb3aa494f21bc8e506004318650 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:03.712028 16919 leader_election.cc:304] T 3cb85eb3aa494f21bc8e506004318650 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97df55a0361841319adfae27c9787b2f; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88
I20260929 02:01:03.712505 17337 raft_consensus.cc:2749] T 3cb85eb3aa494f21bc8e506004318650 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
W20260929 02:01:03.713223 16919 leader_election.cc:343] T 0aa899b966dd481088d18119ee57d81a P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:03.713335 16919 leader_election.cc:304] T 0aa899b966dd481088d18119ee57d81a P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97df55a0361841319adfae27c9787b2f; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88
I20260929 02:01:03.713549 17337 raft_consensus.cc:2749] T 0aa899b966dd481088d18119ee57d81a P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:03.712028 17330 ts_tablet_manager.cc:1480] T ca4fa192c95745bfa163ef63f98d29a5 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.152s user 0.035s sys 0.088s
I20260929 02:01:03.714215 17330 tablet_bootstrap.cc:492] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:03.716038 17330 tablet_bootstrap.cc:654] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.716361 17338 raft_consensus.cc:493] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:03.716559 17338 raft_consensus.cc:515] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.717344 17338 leader_election.cc:290] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:03.717979 16985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2bc3dbd1c82e4d2cb9577c51d9b19224" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
W20260929 02:01:03.718539 17056 leader_election.cc:343] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:03.718869 17334 tablet_bootstrap.cc:492] T 43f983a2802b4ad3871d743b42b5deea P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:03.719031 17335 tablet_bootstrap.cc:492] T b591026c764b48a9af227b7be6dd0ee6 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:03.719045 17334 ts_tablet_manager.cc:1449] T 43f983a2802b4ad3871d743b42b5deea P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.161s user 0.063s sys 0.072s
I20260929 02:01:03.719205 17335 ts_tablet_manager.cc:1449] T b591026c764b48a9af227b7be6dd0ee6 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.183s user 0.068s sys 0.105s
I20260929 02:01:03.719159 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a0e71f5f586410fabe38e3c3b26dfd0" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:03.719242 17330 tablet_bootstrap.cc:492] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:03.719385 17330 ts_tablet_manager.cc:1449] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 02:01:03.719534 17253 raft_consensus.cc:2468] T 9a0e71f5f586410fabe38e3c3b26dfd0 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:03.719970 17334 raft_consensus.cc:359] T 43f983a2802b4ad3871d743b42b5deea P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.720028 17335 raft_consensus.cc:359] T b591026c764b48a9af227b7be6dd0ee6 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.720142 17334 raft_consensus.cc:385] T 43f983a2802b4ad3871d743b42b5deea P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.720218 17335 raft_consensus.cc:385] T b591026c764b48a9af227b7be6dd0ee6 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.720237 17334 raft_consensus.cc:740] T 43f983a2802b4ad3871d743b42b5deea P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.720191 17330 raft_consensus.cc:359] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.720333 17335 raft_consensus.cc:740] T b591026c764b48a9af227b7be6dd0ee6 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.720371 17330 raft_consensus.cc:385] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.720546 17330 raft_consensus.cc:740] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.720520 17334 consensus_queue.cc:260] T 43f983a2802b4ad3871d743b42b5deea P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.720595 17335 consensus_queue.cc:260] T b591026c764b48a9af227b7be6dd0ee6 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.720719 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9103285f38f149a8ab80fdd0eb792dee" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:03.720817 17330 consensus_queue.cc:260] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.721055 17334 ts_tablet_manager.cc:1480] T 43f983a2802b4ad3871d743b42b5deea P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 02:01:03.721113 17335 ts_tablet_manager.cc:1480] T b591026c764b48a9af227b7be6dd0ee6 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260929 02:01:03.721280 17330 ts_tablet_manager.cc:1480] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 02:01:03.721376 17335 tablet_bootstrap.cc:492] T 8601b0ded51b49d0abeb95f7e5fedf8c P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:03.721362 17252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2bc3dbd1c82e4d2cb9577c51d9b19224" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:03.722740 17334 tablet_bootstrap.cc:492] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:03.722844 17330 tablet_bootstrap.cc:492] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:03.723160 17335 tablet_bootstrap.cc:654] T 8601b0ded51b49d0abeb95f7e5fedf8c P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.723591 17348 raft_consensus.cc:493] T 43f983a2802b4ad3871d743b42b5deea P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:03.723749 17348 raft_consensus.cc:515] T 43f983a2802b4ad3871d743b42b5deea P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.725710 17348 leader_election.cc:290] T 43f983a2802b4ad3871d743b42b5deea P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:03.726099 17330 tablet_bootstrap.cc:654] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.728960 17334 tablet_bootstrap.cc:654] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.720227 17055 leader_election.cc:304] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters: 97df55a0361841319adfae27c9787b2f
I20260929 02:01:03.737371 17338 raft_consensus.cc:2804] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:03.737546 17338 raft_consensus.cc:493] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:03.737661 17338 raft_consensus.cc:3060] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
W20260929 02:01:03.741166 17055 leader_election.cc:343] T 9103285f38f149a8ab80fdd0eb792dee P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:03.741372 17055 leader_election.cc:304] T 9103285f38f149a8ab80fdd0eb792dee P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc; no voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88
I20260929 02:01:03.744964 17338 raft_consensus.cc:515] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.746155 17338 leader_election.cc:290] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:03.746466 17338 raft_consensus.cc:2749] T 9103285f38f149a8ab80fdd0eb792dee P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:03.746837 16985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a0e71f5f586410fabe38e3c3b26dfd0" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
W20260929 02:01:03.747596 17056 leader_election.cc:343] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:03.751430 17334 tablet_bootstrap.cc:492] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:03.751680 17334 ts_tablet_manager.cc:1449] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.029s user 0.003s sys 0.006s
I20260929 02:01:03.755908 17334 raft_consensus.cc:359] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.756191 17334 raft_consensus.cc:385] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.756300 17334 raft_consensus.cc:740] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.762574 17334 consensus_queue.cc:260] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.762866 17348 raft_consensus.cc:493] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:03.763047 17348 raft_consensus.cc:515] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.764864 17348 leader_election.cc:290] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:03.768812 17334 ts_tablet_manager.cc:1480] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.017s user 0.004s sys 0.000s
W20260929 02:01:03.771673 17055 leader_election.cc:343] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:03.771883 17055 leader_election.cc:304] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc; no voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88
I20260929 02:01:03.773209 17252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a0e71f5f586410fabe38e3c3b26dfd0" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:03.773478 17252 raft_consensus.cc:3060] T 9a0e71f5f586410fabe38e3c3b26dfd0 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:03.774204 17334 tablet_bootstrap.cc:492] T b591026c764b48a9af227b7be6dd0ee6 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:03.778601 17252 raft_consensus.cc:2468] T 9a0e71f5f586410fabe38e3c3b26dfd0 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:03.781992 17369 raft_consensus.cc:2749] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:03.784792 17055 leader_election.cc:304] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters: 97df55a0361841319adfae27c9787b2f
I20260929 02:01:03.785348 17369 raft_consensus.cc:2804] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:03.785670 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8601b0ded51b49d0abeb95f7e5fedf8c" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:03.785609 16985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f983a2802b4ad3871d743b42b5deea" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:03.785920 16986 raft_consensus.cc:2468] T 8601b0ded51b49d0abeb95f7e5fedf8c P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 0.
I20260929 02:01:03.786286 17369 raft_consensus.cc:697] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Becoming Leader. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Running, Role: LEADER
W20260929 02:01:03.786444 17189 leader_election.cc:343] T 43f983a2802b4ad3871d743b42b5deea P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:03.786805 17189 leader_election.cc:304] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:03.787259 17348 raft_consensus.cc:2804] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:03.787375 17348 raft_consensus.cc:493] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:03.787531 17334 tablet_bootstrap.cc:654] T b591026c764b48a9af227b7be6dd0ee6 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.787513 17348 raft_consensus.cc:3060] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:03.787372 17369 consensus_queue.cc:237] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.791944 17348 raft_consensus.cc:515] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.793500 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8601b0ded51b49d0abeb95f7e5fedf8c" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:03.793809 16986 raft_consensus.cc:3060] T 8601b0ded51b49d0abeb95f7e5fedf8c P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:03.795204 17348 leader_election.cc:290] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:03.796780 17330 tablet_bootstrap.cc:492] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:03.796940 17330 ts_tablet_manager.cc:1449] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.074s user 0.029s sys 0.030s
I20260929 02:01:03.797206 16986 raft_consensus.cc:2468] T 8601b0ded51b49d0abeb95f7e5fedf8c P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 1.
I20260929 02:01:03.797281 17334 tablet_bootstrap.cc:492] T b591026c764b48a9af227b7be6dd0ee6 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:03.797530 17334 ts_tablet_manager.cc:1449] T b591026c764b48a9af227b7be6dd0ee6 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.023s user 0.011s sys 0.000s
I20260929 02:01:03.797714 17189 leader_election.cc:304] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:03.797878 17330 raft_consensus.cc:359] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.798059 17330 raft_consensus.cc:385] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.798106 17348 raft_consensus.cc:2804] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:03.798164 17330 raft_consensus.cc:740] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.798432 17330 consensus_queue.cc:260] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.798724 17334 raft_consensus.cc:359] T b591026c764b48a9af227b7be6dd0ee6 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.798970 17330 ts_tablet_manager.cc:1480] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 02:01:03.799000 17334 raft_consensus.cc:385] T b591026c764b48a9af227b7be6dd0ee6 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.799119 17334 raft_consensus.cc:740] T b591026c764b48a9af227b7be6dd0ee6 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.799322 17330 tablet_bootstrap.cc:492] T 7a962f60729242a1bcebe88c89f8f895 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:03.799005 17348 raft_consensus.cc:697] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Becoming Leader. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Running, Role: LEADER
I20260929 02:01:03.799379 17334 consensus_queue.cc:260] T b591026c764b48a9af227b7be6dd0ee6 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.799896 17334 ts_tablet_manager.cc:1480] T b591026c764b48a9af227b7be6dd0ee6 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 02:01:03.800180 17334 tablet_bootstrap.cc:492] T c64e85b42e0d4618885975ac4125d4f2 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:03.801079 17330 tablet_bootstrap.cc:654] T 7a962f60729242a1bcebe88c89f8f895 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.801872 17334 tablet_bootstrap.cc:654] T c64e85b42e0d4618885975ac4125d4f2 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.802414 17348 consensus_queue.cc:237] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.805208 17344 raft_consensus.cc:493] T b2d556cada174f83a633a7a8836826ce P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:03.805392 17344 raft_consensus.cc:515] T b2d556cada174f83a633a7a8836826ce P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.806541 17344 leader_election.cc:290] T b2d556cada174f83a633a7a8836826ce P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:03.807446 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2d556cada174f83a633a7a8836826ce" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
W20260929 02:01:03.808188 17189 leader_election.cc:343] T b2d556cada174f83a633a7a8836826ce P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:03.808905 17359 raft_consensus.cc:493] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:03.809082 17359 raft_consensus.cc:515] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.810796 17252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7189d73aa7c43b99293f2da0e47338b" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:03.811038 17252 raft_consensus.cc:2468] T c7189d73aa7c43b99293f2da0e47338b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:03.812038 16921 leader_election.cc:304] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:03.812482 17337 raft_consensus.cc:2804] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:03.812606 17337 raft_consensus.cc:493] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:03.812783 17337 raft_consensus.cc:3060] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:03.815650 17335 tablet_bootstrap.cc:492] T 8601b0ded51b49d0abeb95f7e5fedf8c P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:03.815876 17335 ts_tablet_manager.cc:1449] T 8601b0ded51b49d0abeb95f7e5fedf8c P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.095s user 0.017s sys 0.062s
I20260929 02:01:03.817135 17335 raft_consensus.cc:359] T 8601b0ded51b49d0abeb95f7e5fedf8c P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.817374 17335 raft_consensus.cc:385] T 8601b0ded51b49d0abeb95f7e5fedf8c P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.817466 17335 raft_consensus.cc:740] T 8601b0ded51b49d0abeb95f7e5fedf8c P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.817705 17119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7189d73aa7c43b99293f2da0e47338b" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:03.817780 17335 consensus_queue.cc:260] T 8601b0ded51b49d0abeb95f7e5fedf8c P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.818042 17119 raft_consensus.cc:2468] T c7189d73aa7c43b99293f2da0e47338b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:03.818475 17335 ts_tablet_manager.cc:1480] T 8601b0ded51b49d0abeb95f7e5fedf8c P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260929 02:01:03.818851 17335 tablet_bootstrap.cc:492] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:03.820685 17120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8601b0ded51b49d0abeb95f7e5fedf8c" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:03.820883 17335 tablet_bootstrap.cc:654] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.821017 17120 raft_consensus.cc:2468] T 8601b0ded51b49d0abeb95f7e5fedf8c P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 0.
I20260929 02:01:03.821481 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8601b0ded51b49d0abeb95f7e5fedf8c" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:03.821828 17118 raft_consensus.cc:3060] T 8601b0ded51b49d0abeb95f7e5fedf8c P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:03.824692 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2d556cada174f83a633a7a8836826ce" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:03.829289 17118 raft_consensus.cc:2468] T 8601b0ded51b49d0abeb95f7e5fedf8c P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 1.
W20260929 02:01:03.830605 17186 leader_election.cc:343] T b2d556cada174f83a633a7a8836826ce P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:03.830790 17186 leader_election.cc:304] T b2d556cada174f83a633a7a8836826ce P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f
I20260929 02:01:03.831136 17344 raft_consensus.cc:2749] T b2d556cada174f83a633a7a8836826ce P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:03.831905 17359 leader_election.cc:290] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:03.832932 17337 raft_consensus.cc:515] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.835721 17252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7189d73aa7c43b99293f2da0e47338b" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:03.835983 17252 raft_consensus.cc:3060] T c7189d73aa7c43b99293f2da0e47338b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:03.840245 17252 raft_consensus.cc:2468] T c7189d73aa7c43b99293f2da0e47338b P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:03.840873 16921 leader_election.cc:304] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:03.841343 17359 raft_consensus.cc:2804] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:03.841841 17337 leader_election.cc:290] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Requested vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:03.843318 17359 raft_consensus.cc:697] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Becoming Leader. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Running, Role: LEADER
I20260929 02:01:03.844097 17334 tablet_bootstrap.cc:492] T c64e85b42e0d4618885975ac4125d4f2 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:03.844336 17334 ts_tablet_manager.cc:1449] T c64e85b42e0d4618885975ac4125d4f2 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.044s user 0.003s sys 0.000s
I20260929 02:01:03.844378 17359 consensus_queue.cc:237] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.845398 17334 raft_consensus.cc:359] T c64e85b42e0d4618885975ac4125d4f2 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.845690 17334 raft_consensus.cc:385] T c64e85b42e0d4618885975ac4125d4f2 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.845788 17334 raft_consensus.cc:740] T c64e85b42e0d4618885975ac4125d4f2 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.846148 17334 consensus_queue.cc:260] T c64e85b42e0d4618885975ac4125d4f2 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.846889 17334 ts_tablet_manager.cc:1480] T c64e85b42e0d4618885975ac4125d4f2 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.002s user 0.003s sys 0.000s
I20260929 02:01:03.847853 17334 tablet_bootstrap.cc:492] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:03.849155 17330 tablet_bootstrap.cc:492] T 7a962f60729242a1bcebe88c89f8f895 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:03.849341 17330 ts_tablet_manager.cc:1449] T 7a962f60729242a1bcebe88c89f8f895 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.050s user 0.015s sys 0.010s
I20260929 02:01:03.850212 17330 raft_consensus.cc:359] T 7a962f60729242a1bcebe88c89f8f895 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.850298 17334 tablet_bootstrap.cc:654] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.850410 17330 raft_consensus.cc:385] T 7a962f60729242a1bcebe88c89f8f895 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.850503 17330 raft_consensus.cc:740] T 7a962f60729242a1bcebe88c89f8f895 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.850889 17330 consensus_queue.cc:260] T 7a962f60729242a1bcebe88c89f8f895 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.852726 17330 ts_tablet_manager.cc:1480] T 7a962f60729242a1bcebe88c89f8f895 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 02:01:03.854538 17344 raft_consensus.cc:493] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:03.854734 17344 raft_consensus.cc:515] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.854856 17330 tablet_bootstrap.cc:492] T b2d556cada174f83a633a7a8836826ce P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:03.855019 16819 catalog_manager.cc:5838] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97df55a0361841319adfae27c9787b2f (127.0.71.65). New cstate: current_term: 1 leader_uuid: "97df55a0361841319adfae27c9787b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:03.855436 17344 leader_election.cc:290] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:03.856177 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca4fa192c95745bfa163ef63f98d29a5" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:03.856411 16986 raft_consensus.cc:2468] T ca4fa192c95745bfa163ef63f98d29a5 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 0.
I20260929 02:01:03.856961 17189 leader_election.cc:304] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:03.857376 17330 tablet_bootstrap.cc:654] T b2d556cada174f83a633a7a8836826ce P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.857386 17344 raft_consensus.cc:2804] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:03.857494 17344 raft_consensus.cc:493] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:03.857587 17344 raft_consensus.cc:3060] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:03.861403 17344 raft_consensus.cc:515] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.862460 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca4fa192c95745bfa163ef63f98d29a5" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:03.863186 17119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f983a2802b4ad3871d743b42b5deea" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:03.863571 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7189d73aa7c43b99293f2da0e47338b" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:03.863787 17330 tablet_bootstrap.cc:492] T b2d556cada174f83a633a7a8836826ce P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:03.863819 17118 raft_consensus.cc:3060] T c7189d73aa7c43b99293f2da0e47338b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:03.863976 17330 ts_tablet_manager.cc:1449] T b2d556cada174f83a633a7a8836826ce P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.009s user 0.003s sys 0.003s
I20260929 02:01:03.864921 17330 raft_consensus.cc:359] T b2d556cada174f83a633a7a8836826ce P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.865087 17330 raft_consensus.cc:385] T b2d556cada174f83a633a7a8836826ce P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
W20260929 02:01:03.865075 17186 leader_election.cc:343] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:03.865173 17330 raft_consensus.cc:740] T b2d556cada174f83a633a7a8836826ce P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
W20260929 02:01:03.865422 17186 leader_election.cc:343] T 43f983a2802b4ad3871d743b42b5deea P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:03.865415 17330 consensus_queue.cc:260] T b2d556cada174f83a633a7a8836826ce P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.865562 17186 leader_election.cc:304] T 43f983a2802b4ad3871d743b42b5deea P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f
I20260929 02:01:03.865607 16818 catalog_manager.cc:5838] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66). New cstate: current_term: 1 leader_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:03.865960 17330 ts_tablet_manager.cc:1480] T b2d556cada174f83a633a7a8836826ce P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 02:01:03.866240 17330 tablet_bootstrap.cc:492] T 37e07f15e43c49a4bf5ab1334dd85d22 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:03.866797 17118 raft_consensus.cc:2468] T c7189d73aa7c43b99293f2da0e47338b P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:03.867254 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca4fa192c95745bfa163ef63f98d29a5" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:03.867520 16986 raft_consensus.cc:3060] T ca4fa192c95745bfa163ef63f98d29a5 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:03.868719 16817 catalog_manager.cc:5838] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88 reported cstate change: term changed from 0 to 1, leader changed from <none> to d103dd4823044faebe46100a0dfbee88 (127.0.71.67). New cstate: current_term: 1 leader_uuid: "d103dd4823044faebe46100a0dfbee88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: HEALTHY } } }
I20260929 02:01:03.870342 16986 raft_consensus.cc:2468] T ca4fa192c95745bfa163ef63f98d29a5 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 1.
I20260929 02:01:03.870447 17334 tablet_bootstrap.cc:492] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:03.870587 17334 ts_tablet_manager.cc:1449] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.023s user 0.008s sys 0.003s
I20260929 02:01:03.870836 17189 leader_election.cc:304] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:03.871470 17334 raft_consensus.cc:359] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.871668 17334 raft_consensus.cc:385] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.871793 17344 leader_election.cc:290] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:03.872026 17335 tablet_bootstrap.cc:492] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:03.872192 17335 ts_tablet_manager.cc:1449] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.053s user 0.016s sys 0.015s
I20260929 02:01:03.872280 17330 tablet_bootstrap.cc:654] T 37e07f15e43c49a4bf5ab1334dd85d22 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.872399 17348 raft_consensus.cc:2804] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:03.872716 17348 raft_consensus.cc:697] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Becoming Leader. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Running, Role: LEADER
I20260929 02:01:03.873111 17335 raft_consensus.cc:359] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.872226 17344 raft_consensus.cc:2749] T 43f983a2802b4ad3871d743b42b5deea P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:03.873319 17335 raft_consensus.cc:385] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.873427 17335 raft_consensus.cc:740] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.873660 17335 consensus_queue.cc:260] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.873145 17348 consensus_queue.cc:237] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.871804 17334 raft_consensus.cc:740] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.875257 17334 consensus_queue.cc:260] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.875962 17334 ts_tablet_manager.cc:1480] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260929 02:01:03.876665 17334 tablet_bootstrap.cc:492] T 3cb85eb3aa494f21bc8e506004318650 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:03.877866 17335 ts_tablet_manager.cc:1480] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260929 02:01:03.878629 17334 tablet_bootstrap.cc:654] T 3cb85eb3aa494f21bc8e506004318650 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.881503 17330 tablet_bootstrap.cc:492] T 37e07f15e43c49a4bf5ab1334dd85d22 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:03.881726 17330 ts_tablet_manager.cc:1449] T 37e07f15e43c49a4bf5ab1334dd85d22 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.016s user 0.005s sys 0.001s
I20260929 02:01:03.882773 17330 raft_consensus.cc:359] T 37e07f15e43c49a4bf5ab1334dd85d22 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.883047 17330 raft_consensus.cc:385] T 37e07f15e43c49a4bf5ab1334dd85d22 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.883147 17330 raft_consensus.cc:740] T 37e07f15e43c49a4bf5ab1334dd85d22 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.883450 17330 consensus_queue.cc:260] T 37e07f15e43c49a4bf5ab1334dd85d22 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.884153 17330 ts_tablet_manager.cc:1480] T 37e07f15e43c49a4bf5ab1334dd85d22 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 02:01:03.884558 17330 tablet_bootstrap.cc:492] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:03.886973 17330 tablet_bootstrap.cc:654] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.890223 17330 tablet_bootstrap.cc:492] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:03.890417 17330 ts_tablet_manager.cc:1449] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 02:01:03.891145 17330 raft_consensus.cc:359] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.891394 17330 raft_consensus.cc:385] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.891513 17330 raft_consensus.cc:740] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.891831 17330 consensus_queue.cc:260] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.892596 17330 ts_tablet_manager.cc:1480] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260929 02:01:03.892946 17330 tablet_bootstrap.cc:492] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:03.895224 17330 tablet_bootstrap.cc:654] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.896536 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca4fa192c95745bfa163ef63f98d29a5" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
W20260929 02:01:03.897333 17186 leader_election.cc:343] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:03.899245 17335 tablet_bootstrap.cc:492] T 43f983a2802b4ad3871d743b42b5deea P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:03.899293 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:03.899609 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:03.899902 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:03.900979 17335 tablet_bootstrap.cc:654] T 43f983a2802b4ad3871d743b42b5deea P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.901494 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:03.905761 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.002s sys 0.002s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":184,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:03.905856 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":5858}
I20260929 02:01:03.909176 17335 tablet_bootstrap.cc:492] T 43f983a2802b4ad3871d743b42b5deea P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:03.909405 17335 ts_tablet_manager.cc:1449] T 43f983a2802b4ad3871d743b42b5deea P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.001s
I20260929 02:01:03.910461 17335 raft_consensus.cc:359] T 43f983a2802b4ad3871d743b42b5deea P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.910768 17335 raft_consensus.cc:385] T 43f983a2802b4ad3871d743b42b5deea P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.910974 17335 raft_consensus.cc:740] T 43f983a2802b4ad3871d743b42b5deea P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.911329 17335 consensus_queue.cc:260] T 43f983a2802b4ad3871d743b42b5deea P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.912559 17335 ts_tablet_manager.cc:1480] T 43f983a2802b4ad3871d743b42b5deea P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260929 02:01:03.912997 17335 tablet_bootstrap.cc:492] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:03.915318 17335 tablet_bootstrap.cc:654] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.921030 17359 raft_consensus.cc:493] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:03.921209 17359 raft_consensus.cc:515] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.921931 17359 leader_election.cc:290] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:03.923017 17330 tablet_bootstrap.cc:492] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:03.922988 17252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b591026c764b48a9af227b7be6dd0ee6" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:03.923190 17330 ts_tablet_manager.cc:1449] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.030s user 0.010s sys 0.015s
I20260929 02:01:03.923205 17252 raft_consensus.cc:2468] T b591026c764b48a9af227b7be6dd0ee6 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:03.924021 17330 raft_consensus.cc:359] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.924201 17330 raft_consensus.cc:385] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.924291 17330 raft_consensus.cc:740] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.924567 17330 consensus_queue.cc:260] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.925096 17330 ts_tablet_manager.cc:1480] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:01:03.925841 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b591026c764b48a9af227b7be6dd0ee6" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:03.926013 16921 leader_election.cc:304] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:03.926077 17118 raft_consensus.cc:2468] T b591026c764b48a9af227b7be6dd0ee6 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:03.926389 17330 tablet_bootstrap.cc:492] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:03.926400 17337 raft_consensus.cc:2804] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:03.926651 17337 raft_consensus.cc:493] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:03.926738 17337 raft_consensus.cc:3060] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:03.928506 17335 tablet_bootstrap.cc:492] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:03.928696 17335 ts_tablet_manager.cc:1449] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.016s user 0.001s sys 0.003s
I20260929 02:01:03.929522 17335 raft_consensus.cc:359] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.929738 17335 raft_consensus.cc:385] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.929896 17335 raft_consensus.cc:740] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.930236 17335 consensus_queue.cc:260] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.931267 17335 ts_tablet_manager.cc:1480] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:01:03.931660 17369 raft_consensus.cc:493] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:03.931718 17335 tablet_bootstrap.cc:492] T 0aa899b966dd481088d18119ee57d81a P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:03.931830 17369 raft_consensus.cc:515] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.932602 17369 leader_election.cc:290] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:03.932907 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:03.933279 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:03.933748 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:03.933866 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a962f60729242a1bcebe88c89f8f895" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:03.934052 16986 raft_consensus.cc:2468] T 7a962f60729242a1bcebe88c89f8f895 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:03.934128 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:03.929790 17337 raft_consensus.cc:515] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.934669 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:03.935304 17252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a962f60729242a1bcebe88c89f8f895" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:03.935575 17252 raft_consensus.cc:2468] T 7a962f60729242a1bcebe88c89f8f895 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:03.936141 17055 leader_election.cc:304] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:03.937309 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:03.937583 17330 tablet_bootstrap.cc:654] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.937630 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:03.937753 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b591026c764b48a9af227b7be6dd0ee6" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:03.937922 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:03.937942 17118 raft_consensus.cc:3060] T b591026c764b48a9af227b7be6dd0ee6 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:03.938200 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:03.938519 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:03.938861 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:03.939127 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:03.939630 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.317380
I20260929 02:01:03.939970 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.317380
I20260929 02:01:03.940222 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:03.940476 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:03.940773 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:03.942083 17335 tablet_bootstrap.cc:654] T 0aa899b966dd481088d18119ee57d81a P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.944154 17369 raft_consensus.cc:2804] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:03.944283 17369 raft_consensus.cc:493] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:03.944419 17369 raft_consensus.cc:3060] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:03.947454 17369 raft_consensus.cc:515] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.948632 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.010s user 0.005s sys 0.000s Metrics: {"bytes_written":465,"delete_count":0,"lbm_write_time_us":139,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":2}
I20260929 02:01:03.949049 17252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b591026c764b48a9af227b7be6dd0ee6" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:03.949308 17252 raft_consensus.cc:3060] T b591026c764b48a9af227b7be6dd0ee6 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:03.951385 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.018s user 0.002s sys 0.003s Metrics: {"bytes_written":492,"delete_count":0,"lbm_write_time_us":165,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":3}
I20260929 02:01:03.953248 17118 raft_consensus.cc:2468] T b591026c764b48a9af227b7be6dd0ee6 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:03.948859 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a962f60729242a1bcebe88c89f8f895" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:03.954171 16986 raft_consensus.cc:3060] T 7a962f60729242a1bcebe88c89f8f895 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:03.953900 16919 leader_election.cc:304] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:03.956825 17359 raft_consensus.cc:2804] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:03.956984 17359 raft_consensus.cc:697] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Becoming Leader. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Running, Role: LEADER
I20260929 02:01:03.957275 17359 consensus_queue.cc:237] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.957593 16986 raft_consensus.cc:2468] T 7a962f60729242a1bcebe88c89f8f895 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:03.943925 17337 leader_election.cc:290] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Requested vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:03.952332 17252 raft_consensus.cc:2468] T b591026c764b48a9af227b7be6dd0ee6 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:03.958763 17056 leader_election.cc:304] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:03.959161 17338 raft_consensus.cc:2804] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:03.962832 17252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7a962f60729242a1bcebe88c89f8f895" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:03.963079 17252 raft_consensus.cc:3060] T 7a962f60729242a1bcebe88c89f8f895 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:03.973922 17369 leader_election.cc:290] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:03.974277 17369 raft_consensus.cc:493] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:03.974453 17369 raft_consensus.cc:515] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.976090 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2d556cada174f83a633a7a8836826ce" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:03.976434 17253 raft_consensus.cc:2468] T b2d556cada174f83a633a7a8836826ce P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:03.977161 17055 leader_election.cc:304] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:03.977707 17369 leader_election.cc:290] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:03.977886 17369 raft_consensus.cc:2804] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:03.977996 17369 raft_consensus.cc:493] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:03.978096 17369 raft_consensus.cc:3060] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:03.987162 17359 raft_consensus.cc:493] T c3686e82fe0f4b6790a9a7f290c06876 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:03.987376 17359 raft_consensus.cc:515] T c3686e82fe0f4b6790a9a7f290c06876 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } }
I20260929 02:01:03.988081 17252 raft_consensus.cc:2468] T 7a962f60729242a1bcebe88c89f8f895 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:03.988726 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3686e82fe0f4b6790a9a7f290c06876" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
W20260929 02:01:03.989413 16921 leader_election.cc:343] T c3686e82fe0f4b6790a9a7f290c06876 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:03.990594 17359 leader_election.cc:290] T c3686e82fe0f4b6790a9a7f290c06876 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:03.990532 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3686e82fe0f4b6790a9a7f290c06876" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
W20260929 02:01:03.990976 17189 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.71.65:43625, user_credentials={real_user=slave}} blocked reactor thread for 120149us
W20260929 02:01:03.991147 16919 leader_election.cc:343] T c3686e82fe0f4b6790a9a7f290c06876 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:03.991291 16919 leader_election.cc:304] T c3686e82fe0f4b6790a9a7f290c06876 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97df55a0361841319adfae27c9787b2f; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88
I20260929 02:01:03.992260 17359 raft_consensus.cc:2749] T c3686e82fe0f4b6790a9a7f290c06876 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:03.993201 17334 tablet_bootstrap.cc:492] T 3cb85eb3aa494f21bc8e506004318650 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:03.993402 17334 ts_tablet_manager.cc:1449] T 3cb85eb3aa494f21bc8e506004318650 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.117s user 0.029s sys 0.029s
I20260929 02:01:03.994292 17334 raft_consensus.cc:359] T 3cb85eb3aa494f21bc8e506004318650 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.994414 17330 tablet_bootstrap.cc:492] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:03.994498 17334 raft_consensus.cc:385] T 3cb85eb3aa494f21bc8e506004318650 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.994560 17330 ts_tablet_manager.cc:1449] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.068s user 0.006s sys 0.023s
I20260929 02:01:03.994570 17334 raft_consensus.cc:740] T 3cb85eb3aa494f21bc8e506004318650 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.995391 17330 raft_consensus.cc:359] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.995577 17330 raft_consensus.cc:385] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:03.995669 17330 raft_consensus.cc:740] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:03.995901 17330 consensus_queue.cc:260] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.995949 17334 consensus_queue.cc:260] T 3cb85eb3aa494f21bc8e506004318650 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.996811 17334 ts_tablet_manager.cc:1480] T 3cb85eb3aa494f21bc8e506004318650 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:01:03.997251 17334 tablet_bootstrap.cc:492] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:03.998090 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2d556cada174f83a633a7a8836826ce" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:03.998334 16986 raft_consensus.cc:2468] T b2d556cada174f83a633a7a8836826ce P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:03.998317 17369 raft_consensus.cc:515] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:03.998646 17330 ts_tablet_manager.cc:1480] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260929 02:01:03.998955 17330 tablet_bootstrap.cc:492] T 4d9e051c702b42bc9392c5dcae42265d P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:03.999265 17369 leader_election.cc:290] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:03.999791 17334 tablet_bootstrap.cc:654] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:03.999818 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2d556cada174f83a633a7a8836826ce" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:04.000046 17253 raft_consensus.cc:3060] T b2d556cada174f83a633a7a8836826ce P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.000829 17330 tablet_bootstrap.cc:654] T 4d9e051c702b42bc9392c5dcae42265d P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:04.002027 17335 tablet_bootstrap.cc:492] T 0aa899b966dd481088d18119ee57d81a P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:04.002588 17335 ts_tablet_manager.cc:1449] T 0aa899b966dd481088d18119ee57d81a P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.071s user 0.010s sys 0.019s
I20260929 02:01:04.005302 17253 raft_consensus.cc:2468] T b2d556cada174f83a633a7a8836826ce P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:04.006105 17055 leader_election.cc:304] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:04.007030 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b2d556cada174f83a633a7a8836826ce" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:04.007335 16986 raft_consensus.cc:3060] T b2d556cada174f83a633a7a8836826ce P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.008816 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.069s user 0.017s sys 0.002s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":6667,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":16291,"dirs.run_cpu_time_us":348,"dirs.run_wall_time_us":7204,"lbm_read_time_us":508,"lbm_reads_lt_1ms":23,"lbm_write_time_us":370,"lbm_writes_lt_1ms":9,"mutex_wait_us":3494,"peak_mem_usage":32768,"reinsert_count":0,"thread_start_us":25471,"threads_started":2,"update_count":3}
I20260929 02:01:04.009274 17359 raft_consensus.cc:493] T 37e07f15e43c49a4bf5ab1334dd85d22 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:04.009441 17359 raft_consensus.cc:515] T 37e07f15e43c49a4bf5ab1334dd85d22 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.010321 17359 leader_election.cc:290] T 37e07f15e43c49a4bf5ab1334dd85d22 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:04.012465 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37e07f15e43c49a4bf5ab1334dd85d22" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
W20260929 02:01:04.013490 16919 leader_election.cc:343] T 37e07f15e43c49a4bf5ab1334dd85d22 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:04.015890 16986 raft_consensus.cc:2468] T b2d556cada174f83a633a7a8836826ce P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:04.016623 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37e07f15e43c49a4bf5ab1334dd85d22" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
W20260929 02:01:04.017490 16921 leader_election.cc:343] T 37e07f15e43c49a4bf5ab1334dd85d22 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:04.017673 16921 leader_election.cc:304] T 37e07f15e43c49a4bf5ab1334dd85d22 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97df55a0361841319adfae27c9787b2f; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88
I20260929 02:01:04.018152 17359 raft_consensus.cc:2749] T 37e07f15e43c49a4bf5ab1334dd85d22 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:04.019315 17335 raft_consensus.cc:359] T 0aa899b966dd481088d18119ee57d81a P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.019447 17338 raft_consensus.cc:697] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Becoming Leader. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Running, Role: LEADER
I20260929 02:01:04.019584 17335 raft_consensus.cc:385] T 0aa899b966dd481088d18119ee57d81a P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:04.019686 17335 raft_consensus.cc:740] T 0aa899b966dd481088d18119ee57d81a P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:04.019780 17338 consensus_queue.cc:237] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.020043 17335 consensus_queue.cc:260] T 0aa899b966dd481088d18119ee57d81a P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.020744 17335 ts_tablet_manager.cc:1480] T 0aa899b966dd481088d18119ee57d81a P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.018s user 0.001s sys 0.001s
I20260929 02:01:04.021189 17335 tablet_bootstrap.cc:492] T 37e07f15e43c49a4bf5ab1334dd85d22 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:04.023706 17335 tablet_bootstrap.cc:654] T 37e07f15e43c49a4bf5ab1334dd85d22 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:04.024430 16818 catalog_manager.cc:5838] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88 reported cstate change: term changed from 0 to 1, leader changed from <none> to d103dd4823044faebe46100a0dfbee88 (127.0.71.67). New cstate: current_term: 1 leader_uuid: "d103dd4823044faebe46100a0dfbee88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: HEALTHY } } }
I20260929 02:01:04.027400 17334 tablet_bootstrap.cc:492] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:04.027561 17334 ts_tablet_manager.cc:1449] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.030s user 0.013s sys 0.006s
I20260929 02:01:04.028055 17335 tablet_bootstrap.cc:492] T 37e07f15e43c49a4bf5ab1334dd85d22 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:04.028227 17335 ts_tablet_manager.cc:1449] T 37e07f15e43c49a4bf5ab1334dd85d22 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.007s user 0.002s sys 0.003s
I20260929 02:01:04.028367 17334 raft_consensus.cc:359] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.028800 17369 raft_consensus.cc:2804] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:04.028951 17369 raft_consensus.cc:697] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Becoming Leader. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Running, Role: LEADER
I20260929 02:01:04.029073 17335 raft_consensus.cc:359] T 37e07f15e43c49a4bf5ab1334dd85d22 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.028626 17334 raft_consensus.cc:385] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:04.029246 17335 raft_consensus.cc:385] T 37e07f15e43c49a4bf5ab1334dd85d22 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:04.029320 17334 raft_consensus.cc:740] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:04.029358 17335 raft_consensus.cc:740] T 37e07f15e43c49a4bf5ab1334dd85d22 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:04.029244 17369 consensus_queue.cc:237] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.029561 17334 consensus_queue.cc:260] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.029605 17335 consensus_queue.cc:260] T 37e07f15e43c49a4bf5ab1334dd85d22 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.030231 17334 ts_tablet_manager.cc:1480] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:01:04.030292 17335 ts_tablet_manager.cc:1480] T 37e07f15e43c49a4bf5ab1334dd85d22 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:01:04.030507 17334 tablet_bootstrap.cc:492] T 4d9e051c702b42bc9392c5dcae42265d P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:04.030661 17335 tablet_bootstrap.cc:492] T c64e85b42e0d4618885975ac4125d4f2 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:04.032317 17334 tablet_bootstrap.cc:654] T 4d9e051c702b42bc9392c5dcae42265d P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:04.033968 17330 tablet_bootstrap.cc:492] T 4d9e051c702b42bc9392c5dcae42265d P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:04.034188 17330 ts_tablet_manager.cc:1449] T 4d9e051c702b42bc9392c5dcae42265d P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.013s
I20260929 02:01:04.035149 17330 raft_consensus.cc:359] T 4d9e051c702b42bc9392c5dcae42265d P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.035439 17330 raft_consensus.cc:385] T 4d9e051c702b42bc9392c5dcae42265d P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:04.035534 17330 raft_consensus.cc:740] T 4d9e051c702b42bc9392c5dcae42265d P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:04.035856 17330 consensus_queue.cc:260] T 4d9e051c702b42bc9392c5dcae42265d P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.038255 17330 ts_tablet_manager.cc:1480] T 4d9e051c702b42bc9392c5dcae42265d P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260929 02:01:04.038652 17330 tablet_bootstrap.cc:492] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:04.041111 17330 tablet_bootstrap.cc:654] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:04.042956 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.043273 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.043368 17335 tablet_bootstrap.cc:654] T c64e85b42e0d4618885975ac4125d4f2 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:04.043599 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.043874 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.045926 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.046198 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.046439 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.048769 16819 catalog_manager.cc:5838] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97df55a0361841319adfae27c9787b2f (127.0.71.65). New cstate: current_term: 1 leader_uuid: "97df55a0361841319adfae27c9787b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:04.049890 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.050212 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.050503 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.050725 17330 tablet_bootstrap.cc:492] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:04.050805 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.050917 17330 ts_tablet_manager.cc:1449] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.000s
I20260929 02:01:04.052022 17330 raft_consensus.cc:359] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.052255 17330 raft_consensus.cc:385] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:04.052358 17330 raft_consensus.cc:740] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:04.052670 17330 consensus_queue.cc:260] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.052771 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.053041 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.053258 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.054039 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.222399
I20260929 02:01:04.054340 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.222399
I20260929 02:01:04.054618 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.222399
I20260929 02:01:04.055791 17330 ts_tablet_manager.cc:1480] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.005s user 0.004s sys 0.000s
I20260929 02:01:04.056224 17359 raft_consensus.cc:493] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:04.056362 17359 raft_consensus.cc:515] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.057066 17359 leader_election.cc:290] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:04.058537 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.058763 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.059986 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.016s user 0.002s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":170,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:04.060550 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64e85b42e0d4618885975ac4125d4f2" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:04.061873 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64e85b42e0d4618885975ac4125d4f2" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:04.062106 17253 raft_consensus.cc:2468] T c64e85b42e0d4618885975ac4125d4f2 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
W20260929 02:01:04.063580 16919 leader_election.cc:343] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:04.064231 16921 leader_election.cc:304] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc
I20260929 02:01:04.064798 17359 raft_consensus.cc:2804] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:04.064915 17359 raft_consensus.cc:493] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:04.065009 17359 raft_consensus.cc:3060] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.067848 17359 raft_consensus.cc:515] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.069159 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64e85b42e0d4618885975ac4125d4f2" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
W20260929 02:01:04.069860 16919 leader_election.cc:343] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:04.070830 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64e85b42e0d4618885975ac4125d4f2" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:04.071043 17253 raft_consensus.cc:3060] T c64e85b42e0d4618885975ac4125d4f2 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.071051 17376 raft_consensus.cc:493] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:04.071189 17376 raft_consensus.cc:515] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.071934 17376 leader_election.cc:290] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:04.072568 17252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f983a2802b4ad3871d743b42b5deea" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:04.072779 17252 raft_consensus.cc:2468] T 43f983a2802b4ad3871d743b42b5deea P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:04.072907 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:04.073242 16921 leader_election.cc:304] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:04.073508 17376 raft_consensus.cc:2804] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:04.073534 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.073621 17376 raft_consensus.cc:493] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:04.073704 17376 raft_consensus.cc:3060] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.075510 17253 raft_consensus.cc:2468] T c64e85b42e0d4618885975ac4125d4f2 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:04.076045 16921 leader_election.cc:304] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc
I20260929 02:01:04.076354 17337 raft_consensus.cc:2804] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:04.077153 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f983a2802b4ad3871d743b42b5deea" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:04.077435 17118 raft_consensus.cc:2468] T 43f983a2802b4ad3871d743b42b5deea P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:04.078302 17376 raft_consensus.cc:515] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.079057 17376 leader_election.cc:290] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Requested vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:04.079804 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f983a2802b4ad3871d743b42b5deea" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:04.080010 17118 raft_consensus.cc:3060] T 43f983a2802b4ad3871d743b42b5deea P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.079898 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "43f983a2802b4ad3871d743b42b5deea" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:04.080299 17253 raft_consensus.cc:3060] T 43f983a2802b4ad3871d743b42b5deea P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.083227 17118 raft_consensus.cc:2468] T 43f983a2802b4ad3871d743b42b5deea P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:04.082429 17359 leader_election.cc:290] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Requested vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:04.084110 16919 leader_election.cc:304] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:04.084481 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.033s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":204,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:04.089654 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.089980 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.090266 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.091089 16818 catalog_manager.cc:5838] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66). New cstate: current_term: 1 leader_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:04.092528 17359 raft_consensus.cc:2804] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:04.093055 17359 raft_consensus.cc:697] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Becoming Leader. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Running, Role: LEADER
I20260929 02:01:04.093624 17359 consensus_queue.cc:237] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.094688 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.095065 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.095350 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.095623 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.097226 17253 raft_consensus.cc:2468] T 43f983a2802b4ad3871d743b42b5deea P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:04.097486 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.097793 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.098042 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.103259 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.124093
I20260929 02:01:04.103552 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.124093
I20260929 02:01:04.103856 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.124093
I20260929 02:01:04.108608 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.108879 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
W20260929 02:01:04.110101 17404 log.cc:927] Time spent T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc: Append to log took a long time: real 0.061s user 0.000s sys 0.000s
I20260929 02:01:04.102465 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.002s sys 0.003s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":142,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:04.112458 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.112831 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.113169 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.117448 17337 raft_consensus.cc:697] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Becoming Leader. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Running, Role: LEADER
I20260929 02:01:04.117991 17337 consensus_queue.cc:237] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.124929 16818 catalog_manager.cc:5838] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97df55a0361841319adfae27c9787b2f (127.0.71.65). New cstate: current_term: 1 leader_uuid: "97df55a0361841319adfae27c9787b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:04.128589 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.024s user 0.008s sys 0.004s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":6749,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":383,"dirs.run_cpu_time_us":272,"dirs.run_wall_time_us":1694,"lbm_read_time_us":563,"lbm_reads_lt_1ms":27,"lbm_write_time_us":5375,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":8,"peak_mem_usage":32768,"reinsert_count":0,"update_count":2}
I20260929 02:01:04.129398 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.011s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":10748}
I20260929 02:01:04.129299 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.016s user 0.002s sys 0.003s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":145,"lbm_writes_lt_1ms":4,"mutex_wait_us":9391,"reinsert_count":0,"spinlock_wait_cycles":896,"update_count":1}
I20260929 02:01:04.129805 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.011s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":11055}
I20260929 02:01:04.129873 17337 raft_consensus.cc:493] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:04.130136 17337 raft_consensus.cc:515] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.130548 16819 catalog_manager.cc:5838] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66). New cstate: current_term: 1 leader_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:04.131356 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a0e71f5f586410fabe38e3c3b26dfd0" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:04.131533 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9a0e71f5f586410fabe38e3c3b26dfd0" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:04.131835 17253 raft_consensus.cc:2393] T 9a0e71f5f586410fabe38e3c3b26dfd0 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97df55a0361841319adfae27c9787b2f in current term 1: Already voted for candidate 8668afa5b5ca4ebdabc812d0789de7bc in this term.
I20260929 02:01:04.132654 17337 leader_election.cc:290] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:04.133160 16919 leader_election.cc:304] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97df55a0361841319adfae27c9787b2f; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88
I20260929 02:01:04.133507 17334 tablet_bootstrap.cc:492] T 4d9e051c702b42bc9392c5dcae42265d P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:04.133626 17337 raft_consensus.cc:3060] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.133797 17334 ts_tablet_manager.cc:1449] T 4d9e051c702b42bc9392c5dcae42265d P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.103s user 0.008s sys 0.003s
I20260929 02:01:04.134668 17334 raft_consensus.cc:359] T 4d9e051c702b42bc9392c5dcae42265d P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.134927 17334 raft_consensus.cc:385] T 4d9e051c702b42bc9392c5dcae42265d P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:04.135041 17334 raft_consensus.cc:740] T 4d9e051c702b42bc9392c5dcae42265d P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:04.135392 17334 consensus_queue.cc:260] T 4d9e051c702b42bc9392c5dcae42265d P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.136993 17334 ts_tablet_manager.cc:1480] T 4d9e051c702b42bc9392c5dcae42265d P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260929 02:01:04.137389 17334 tablet_bootstrap.cc:492] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:04.139396 17334 tablet_bootstrap.cc:654] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:04.141659 17335 tablet_bootstrap.cc:492] T c64e85b42e0d4618885975ac4125d4f2 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:04.141939 17335 ts_tablet_manager.cc:1449] T c64e85b42e0d4618885975ac4125d4f2 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.111s user 0.040s sys 0.051s
I20260929 02:01:04.141513 16818 catalog_manager.cc:5838] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97df55a0361841319adfae27c9787b2f (127.0.71.65). New cstate: current_term: 1 leader_uuid: "97df55a0361841319adfae27c9787b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:04.143111 17335 raft_consensus.cc:359] T c64e85b42e0d4618885975ac4125d4f2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.143400 17335 raft_consensus.cc:385] T c64e85b42e0d4618885975ac4125d4f2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:04.143533 17335 raft_consensus.cc:740] T c64e85b42e0d4618885975ac4125d4f2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:04.143846 17335 consensus_queue.cc:260] T c64e85b42e0d4618885975ac4125d4f2 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.144461 17335 ts_tablet_manager.cc:1480] T c64e85b42e0d4618885975ac4125d4f2 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 02:01:04.144827 17335 tablet_bootstrap.cc:492] T ca4fa192c95745bfa163ef63f98d29a5 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:04.146718 17335 tablet_bootstrap.cc:654] T ca4fa192c95745bfa163ef63f98d29a5 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:04.150413 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.154028 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.154443 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.154552 17335 tablet_bootstrap.cc:492] T ca4fa192c95745bfa163ef63f98d29a5 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:04.154708 17335 ts_tablet_manager.cc:1449] T ca4fa192c95745bfa163ef63f98d29a5 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.010s user 0.003s sys 0.005s
I20260929 02:01:04.154870 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.096s user 0.009s sys 0.007s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5457,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":18044,"dirs.run_cpu_time_us":296,"dirs.run_wall_time_us":4665,"lbm_read_time_us":613,"lbm_reads_lt_1ms":27,"lbm_write_time_us":348,"lbm_writes_lt_1ms":9,"mutex_wait_us":5820,"peak_mem_usage":32768,"reinsert_count":0,"thread_start_us":30999,"threads_started":2,"update_count":4}
I20260929 02:01:04.155491 17335 raft_consensus.cc:359] T ca4fa192c95745bfa163ef63f98d29a5 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.155700 17335 raft_consensus.cc:385] T ca4fa192c95745bfa163ef63f98d29a5 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:04.155805 17335 raft_consensus.cc:740] T ca4fa192c95745bfa163ef63f98d29a5 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:04.156090 17335 consensus_queue.cc:260] T ca4fa192c95745bfa163ef63f98d29a5 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.156687 17335 ts_tablet_manager.cc:1480] T ca4fa192c95745bfa163ef63f98d29a5 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 02:01:04.157038 17335 tablet_bootstrap.cc:492] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:04.158805 17335 tablet_bootstrap.cc:654] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:04.162756 17334 tablet_bootstrap.cc:492] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:04.162976 17334 ts_tablet_manager.cc:1449] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.026s user 0.004s sys 0.004s
I20260929 02:01:04.163794 17337 raft_consensus.cc:2749] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:04.164067 17334 raft_consensus.cc:359] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.164331 17334 raft_consensus.cc:385] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:04.164644 17334 raft_consensus.cc:740] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:04.165006 17334 consensus_queue.cc:260] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.166175 17334 ts_tablet_manager.cc:1480] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260929 02:01:04.166610 17334 tablet_bootstrap.cc:492] T 9103285f38f149a8ab80fdd0eb792dee P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:04.169155 17334 tablet_bootstrap.cc:654] T 9103285f38f149a8ab80fdd0eb792dee P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:04.176767 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.022s user 0.001s sys 0.004s Metrics: {"bytes_written":466,"delete_count":0,"lbm_write_time_us":163,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":2}
I20260929 02:01:04.177006 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.177824 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.234474
I20260929 02:01:04.178129 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.178468 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.178807 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.234474
I20260929 02:01:04.183228 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.092s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":92392}
I20260929 02:01:04.183151 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.093s user 0.005s sys 0.000s Metrics: {"bytes_written":506,"delete_count":0,"lbm_write_time_us":172,"lbm_writes_lt_1ms":4,"mutex_wait_us":62577,"reinsert_count":0,"update_count":4}
I20260929 02:01:04.183168 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.092s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":92271}
I20260929 02:01:04.184513 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.184882 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.184873 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.185166 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.185421 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.188678 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.188912 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.189101 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.191237 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.014s user 0.000s sys 0.004s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":167,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:04.200258 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.014s user 0.001s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":131,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:04.200737 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.162666
I20260929 02:01:04.200984 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.162666
I20260929 02:01:04.201213 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.162666
I20260929 02:01:04.201398 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.162666
I20260929 02:01:04.204849 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.205113 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.205329 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.205638 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:04.206135 17335 tablet_bootstrap.cc:492] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:04.206408 17335 ts_tablet_manager.cc:1449] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.049s user 0.022s sys 0.026s
I20260929 02:01:04.207516 17335 raft_consensus.cc:359] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } }
I20260929 02:01:04.207902 17335 raft_consensus.cc:385] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:04.208055 17335 raft_consensus.cc:740] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:04.208464 17335 consensus_queue.cc:260] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } }
I20260929 02:01:04.208585 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.209136 17335 ts_tablet_manager.cc:1480] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:01:04.209542 17335 tablet_bootstrap.cc:492] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:04.211565 17335 tablet_bootstrap.cc:654] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:04.212723 17337 raft_consensus.cc:493] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:04.212917 17337 raft_consensus.cc:515] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.213649 17337 leader_election.cc:290] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:04.214632 17119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2bc3dbd1c82e4d2cb9577c51d9b19224" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:04.214672 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2bc3dbd1c82e4d2cb9577c51d9b19224" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:04.214884 17119 raft_consensus.cc:2468] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:04.214913 17253 raft_consensus.cc:2468] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:04.215401 16921 leader_election.cc:304] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:04.215374 17382 consensus_queue.cc:1048] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:04.215819 17337 raft_consensus.cc:2804] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:04.215924 17337 raft_consensus.cc:493] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:04.216022 17337 raft_consensus.cc:3060] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.216122 17376 raft_consensus.cc:493] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:04.216274 17376 raft_consensus.cc:515] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.216958 17376 leader_election.cc:290] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:04.218981 17337 raft_consensus.cc:515] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.219702 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9103285f38f149a8ab80fdd0eb792dee" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:04.220223 17335 tablet_bootstrap.cc:492] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:04.220379 17335 ts_tablet_manager.cc:1449] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.011s user 0.005s sys 0.004s
I20260929 02:01:04.220352 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9103285f38f149a8ab80fdd0eb792dee" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
W20260929 02:01:04.220404 16921 leader_election.cc:343] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:04.220602 17118 raft_consensus.cc:2468] T 9103285f38f149a8ab80fdd0eb792dee P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:04.221095 16919 leader_election.cc:304] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters: d103dd4823044faebe46100a0dfbee88
I20260929 02:01:04.221455 17335 raft_consensus.cc:359] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.221642 17335 raft_consensus.cc:385] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:04.221619 17376 raft_consensus.cc:2804] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:04.221647 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2bc3dbd1c82e4d2cb9577c51d9b19224" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:04.221730 17376 raft_consensus.cc:493] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:04.221719 17335 raft_consensus.cc:740] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:04.221843 17376 raft_consensus.cc:3060] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.221856 17253 raft_consensus.cc:3060] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.222082 17335 consensus_queue.cc:260] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.222949 17335 ts_tablet_manager.cc:1480] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 02:01:04.224783 17376 raft_consensus.cc:515] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.225119 17337 leader_election.cc:290] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Requested vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:04.226006 17252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9103285f38f149a8ab80fdd0eb792dee" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:04.226531 17120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9103285f38f149a8ab80fdd0eb792dee" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
W20260929 02:01:04.226610 16921 leader_election.cc:343] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:04.226244 17253 raft_consensus.cc:2468] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:04.226727 17120 raft_consensus.cc:3060] T 9103285f38f149a8ab80fdd0eb792dee P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.227262 16921 leader_election.cc:304] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:04.227789 17337 raft_consensus.cc:2804] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:04.227969 17337 raft_consensus.cc:697] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Becoming Leader. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Running, Role: LEADER
I20260929 02:01:04.228257 17337 consensus_queue.cc:237] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.228855 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2bc3dbd1c82e4d2cb9577c51d9b19224" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:04.229022 17118 raft_consensus.cc:3060] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.231956 17118 raft_consensus.cc:2468] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:04.232928 17359 consensus_queue.cc:1048] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:04.233656 17120 raft_consensus.cc:2468] T 9103285f38f149a8ab80fdd0eb792dee P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:04.234139 16919 leader_election.cc:304] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters: d103dd4823044faebe46100a0dfbee88
I20260929 02:01:04.234521 17359 raft_consensus.cc:2804] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:04.234702 17359 raft_consensus.cc:697] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Becoming Leader. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Running, Role: LEADER
I20260929 02:01:04.235060 17359 consensus_queue.cc:237] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.235227 17376 leader_election.cc:290] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Requested vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:04.236523 17334 tablet_bootstrap.cc:492] T 9103285f38f149a8ab80fdd0eb792dee P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:04.236747 17334 ts_tablet_manager.cc:1449] T 9103285f38f149a8ab80fdd0eb792dee P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.070s user 0.024s sys 0.031s
I20260929 02:01:04.238216 17334 raft_consensus.cc:359] T 9103285f38f149a8ab80fdd0eb792dee P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.240532 16818 catalog_manager.cc:5838] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97df55a0361841319adfae27c9787b2f (127.0.71.65). New cstate: current_term: 1 leader_uuid: "97df55a0361841319adfae27c9787b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:04.247745 17334 raft_consensus.cc:385] T 9103285f38f149a8ab80fdd0eb792dee P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:04.247884 17334 raft_consensus.cc:740] T 9103285f38f149a8ab80fdd0eb792dee P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:04.251119 17334 consensus_queue.cc:260] T 9103285f38f149a8ab80fdd0eb792dee P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.251780 17334 ts_tablet_manager.cc:1480] T 9103285f38f149a8ab80fdd0eb792dee P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20260929 02:01:04.251972 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.073s user 0.006s sys 0.006s Metrics: {"cfile_cache_miss":9,"cfile_cache_miss_bytes":7188,"cfile_init":6,"delete_count":0,"delta_blocks_compacted":3,"delta_iterators_relevant":3,"dirs.queue_time_us":1335,"dirs.run_cpu_time_us":301,"dirs.run_wall_time_us":4268,"lbm_read_time_us":579,"lbm_reads_lt_1ms":33,"lbm_write_time_us":288,"lbm_writes_lt_1ms":9,"mutex_wait_us":15075,"peak_mem_usage":32768,"reinsert_count":0,"update_count":4}
I20260929 02:01:04.252744 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.253037 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.253330 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.253868 17334 tablet_bootstrap.cc:492] T c3686e82fe0f4b6790a9a7f290c06876 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:04.254863 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.255736 17334 tablet_bootstrap.cc:654] T c3686e82fe0f4b6790a9a7f290c06876 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:04.256541 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.004s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.256685 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.257018 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.257342 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.264645 17369 raft_consensus.cc:493] T ca4fa192c95745bfa163ef63f98d29a5 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:04.264863 17369 raft_consensus.cc:515] T ca4fa192c95745bfa163ef63f98d29a5 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.264889 17344 raft_consensus.cc:493] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:04.265125 17344 raft_consensus.cc:515] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.265488 17369 leader_election.cc:290] T ca4fa192c95745bfa163ef63f98d29a5 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:04.266039 17344 leader_election.cc:290] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:04.267234 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca4fa192c95745bfa163ef63f98d29a5" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:04.267541 16986 raft_consensus.cc:2393] T ca4fa192c95745bfa163ef63f98d29a5 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8668afa5b5ca4ebdabc812d0789de7bc in current term 1: Already voted for candidate d103dd4823044faebe46100a0dfbee88 in this term.
I20260929 02:01:04.268007 16985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0aa899b966dd481088d18119ee57d81a" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:04.268062 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ca4fa192c95745bfa163ef63f98d29a5" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:04.268198 16985 raft_consensus.cc:2468] T 0aa899b966dd481088d18119ee57d81a P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 0.
I20260929 02:01:04.268610 17189 leader_election.cc:304] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:04.269235 17056 leader_election.cc:304] T ca4fa192c95745bfa163ef63f98d29a5 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc; no voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88
I20260929 02:01:04.270334 17369 raft_consensus.cc:3060] T ca4fa192c95745bfa163ef63f98d29a5 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.273432 17369 raft_consensus.cc:2749] T ca4fa192c95745bfa163ef63f98d29a5 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:04.274437 17344 raft_consensus.cc:2804] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:04.274577 17344 raft_consensus.cc:493] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:04.274698 17344 raft_consensus.cc:3060] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.277930 17344 raft_consensus.cc:515] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.279404 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.026s user 0.004s sys 0.001s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":154,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:04.281639 16985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0aa899b966dd481088d18119ee57d81a" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:04.281853 16985 raft_consensus.cc:3060] T 0aa899b966dd481088d18119ee57d81a P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.286294 16985 raft_consensus.cc:2468] T 0aa899b966dd481088d18119ee57d81a P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 1.
I20260929 02:01:04.286994 17189 leader_election.cc:304] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:04.287309 17382 raft_consensus.cc:2804] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:04.288733 17334 tablet_bootstrap.cc:492] T c3686e82fe0f4b6790a9a7f290c06876 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:04.288897 17334 ts_tablet_manager.cc:1449] T c3686e82fe0f4b6790a9a7f290c06876 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.035s user 0.005s sys 0.004s
I20260929 02:01:04.289497 17369 consensus_queue.cc:1048] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:04.289736 17334 raft_consensus.cc:359] T c3686e82fe0f4b6790a9a7f290c06876 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } }
I20260929 02:01:04.289901 17334 raft_consensus.cc:385] T c3686e82fe0f4b6790a9a7f290c06876 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:04.289989 17334 raft_consensus.cc:740] T c3686e82fe0f4b6790a9a7f290c06876 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:04.290216 17334 consensus_queue.cc:260] T c3686e82fe0f4b6790a9a7f290c06876 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } }
I20260929 02:01:04.297525 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.040s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":39772}
I20260929 02:01:04.297915 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.040s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":40125}
I20260929 02:01:04.298431 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.041s user 0.005s sys 0.002s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":189,"lbm_writes_lt_1ms":4,"mutex_wait_us":32031,"reinsert_count":0,"update_count":1}
I20260929 02:01:04.299052 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.097s user 0.010s sys 0.003s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5619,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":985,"dirs.run_cpu_time_us":334,"dirs.run_wall_time_us":5622,"lbm_read_time_us":458,"lbm_reads_lt_1ms":27,"lbm_write_time_us":345,"lbm_writes_lt_1ms":9,"mutex_wait_us":152,"peak_mem_usage":32768,"reinsert_count":0,"spinlock_wait_cycles":21888,"update_count":5}
I20260929 02:01:04.304024 17119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0aa899b966dd481088d18119ee57d81a" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:04.304349 17119 raft_consensus.cc:2468] T 0aa899b966dd481088d18119ee57d81a P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 0.
I20260929 02:01:04.309672 17344 leader_election.cc:290] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:04.304541 16818 catalog_manager.cc:5838] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97df55a0361841319adfae27c9787b2f (127.0.71.65). New cstate: current_term: 1 leader_uuid: "97df55a0361841319adfae27c9787b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:04.314154 17119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "0aa899b966dd481088d18119ee57d81a" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:04.314388 17119 raft_consensus.cc:3060] T 0aa899b966dd481088d18119ee57d81a P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.321395 17119 raft_consensus.cc:2468] T 0aa899b966dd481088d18119ee57d81a P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 1.
I20260929 02:01:04.328374 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.328804 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.329088 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.330749 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.331067 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.331329 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.331555 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.333185 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.333406 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.333577 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.334028 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.200636
I20260929 02:01:04.334252 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.200636
I20260929 02:01:04.334260 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.003s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":190,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:04.334494 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.200636
I20260929 02:01:04.334980 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.335202 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.337127 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.337122 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"spinlock_wait_cycles":768}
I20260929 02:01:04.337561 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.120292
I20260929 02:01:04.341004 17369 consensus_queue.cc:1048] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:04.341392 17337 consensus_queue.cc:1048] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:04.343503 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:04.344691 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.013s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":143,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:04.345659 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.345733 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:04.350800 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.013s user 0.006s sys 0.005s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":6971,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":106,"lbm_read_time_us":527,"lbm_reads_lt_1ms":27,"lbm_write_time_us":373,"lbm_writes_lt_1ms":9,"peak_mem_usage":32768,"reinsert_count":0,"update_count":2}
I20260929 02:01:04.351619 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.354902 17344 consensus_queue.cc:1048] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:04.355582 17344 raft_consensus.cc:493] T 9103285f38f149a8ab80fdd0eb792dee P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:04.355834 17344 raft_consensus.cc:515] T 9103285f38f149a8ab80fdd0eb792dee P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.357481 17120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9103285f38f149a8ab80fdd0eb792dee" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:04.357851 17120 raft_consensus.cc:2393] T 9103285f38f149a8ab80fdd0eb792dee P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d103dd4823044faebe46100a0dfbee88 in current term 1: Already voted for candidate 97df55a0361841319adfae27c9787b2f in this term.
I20260929 02:01:04.358220 17344 leader_election.cc:290] T 9103285f38f149a8ab80fdd0eb792dee P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:04.358760 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.024s user 0.012s sys 0.000s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5734,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":1526,"lbm_read_time_us":568,"lbm_reads_lt_1ms":27,"lbm_write_time_us":348,"lbm_writes_lt_1ms":9,"mutex_wait_us":4068,"peak_mem_usage":32768,"reinsert_count":0,"update_count":2}
I20260929 02:01:04.365087 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9103285f38f149a8ab80fdd0eb792dee" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:04.369794 17433 consensus_queue.cc:1048] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:04.374637 17433 raft_consensus.cc:493] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:04.374884 17433 raft_consensus.cc:515] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.376339 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37e07f15e43c49a4bf5ab1334dd85d22" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:04.376672 17118 raft_consensus.cc:2468] T 37e07f15e43c49a4bf5ab1334dd85d22 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 0.
I20260929 02:01:04.377458 17186 leader_election.cc:304] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:04.377756 17348 raft_consensus.cc:2804] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:04.377903 17348 raft_consensus.cc:493] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:04.378007 17348 raft_consensus.cc:3060] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.378374 17433 leader_election.cc:290] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:04.378839 17369 raft_consensus.cc:493] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:04.379035 17369 raft_consensus.cc:515] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.379937 17369 leader_election.cc:290] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:04.381072 17348 raft_consensus.cc:515] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.381533 16985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cb85eb3aa494f21bc8e506004318650" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:04.381647 17348 leader_election.cc:290] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:04.381809 16985 raft_consensus.cc:2468] T 3cb85eb3aa494f21bc8e506004318650 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:04.382288 17382 raft_consensus.cc:697] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Becoming Leader. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Running, Role: LEADER
I20260929 02:01:04.382491 17056 leader_election.cc:304] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:04.382712 17382 consensus_queue.cc:237] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.383172 17338 raft_consensus.cc:2804] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:04.383296 17338 raft_consensus.cc:493] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:04.383414 17338 raft_consensus.cc:3060] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.385890 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37e07f15e43c49a4bf5ab1334dd85d22" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:04.386214 17118 raft_consensus.cc:3060] T 37e07f15e43c49a4bf5ab1334dd85d22 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.386538 17338 raft_consensus.cc:515] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.389079 17338 leader_election.cc:290] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:04.387794 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cb85eb3aa494f21bc8e506004318650" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:04.389825 17251 raft_consensus.cc:2468] T 3cb85eb3aa494f21bc8e506004318650 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:04.389943 17252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cb85eb3aa494f21bc8e506004318650" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:04.390100 17252 raft_consensus.cc:3060] T 3cb85eb3aa494f21bc8e506004318650 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.390653 17338 raft_consensus.cc:493] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:04.390863 17338 raft_consensus.cc:515] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.391887 16985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d9e051c702b42bc9392c5dcae42265d" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:04.392182 16985 raft_consensus.cc:2468] T 4d9e051c702b42bc9392c5dcae42265d P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:04.392740 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d9e051c702b42bc9392c5dcae42265d" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:04.392912 17251 raft_consensus.cc:2468] T 4d9e051c702b42bc9392c5dcae42265d P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:04.393337 17056 leader_election.cc:304] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:04.393628 17398 raft_consensus.cc:493] T c64e85b42e0d4618885975ac4125d4f2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:04.393688 17399 raft_consensus.cc:2804] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:04.393795 17399 raft_consensus.cc:493] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:04.393893 17334 ts_tablet_manager.cc:1480] T c3686e82fe0f4b6790a9a7f290c06876 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.105s user 0.027s sys 0.040s
I20260929 02:01:04.393934 17399 raft_consensus.cc:3060] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.393776 17398 raft_consensus.cc:515] T c64e85b42e0d4618885975ac4125d4f2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.394289 17338 leader_election.cc:290] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:04.394954 17398 leader_election.cc:290] T c64e85b42e0d4618885975ac4125d4f2 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:04.395437 17252 raft_consensus.cc:2468] T 3cb85eb3aa494f21bc8e506004318650 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:04.395531 17118 raft_consensus.cc:2468] T 37e07f15e43c49a4bf5ab1334dd85d22 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 1.
I20260929 02:01:04.395731 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64e85b42e0d4618885975ac4125d4f2" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:04.395838 17055 leader_election.cc:304] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:04.395999 17251 raft_consensus.cc:2393] T c64e85b42e0d4618885975ac4125d4f2 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8668afa5b5ca4ebdabc812d0789de7bc in current term 1: Already voted for candidate 97df55a0361841319adfae27c9787b2f in this term.
I20260929 02:01:04.396255 17398 raft_consensus.cc:2804] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:04.398165 17189 leader_election.cc:304] T 9103285f38f149a8ab80fdd0eb792dee P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f
I20260929 02:01:04.398538 17428 raft_consensus.cc:3060] T 9103285f38f149a8ab80fdd0eb792dee P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.400676 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37e07f15e43c49a4bf5ab1334dd85d22" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:04.400879 16984 raft_consensus.cc:3060] T 37e07f15e43c49a4bf5ab1334dd85d22 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.402971 17428 raft_consensus.cc:2749] T 9103285f38f149a8ab80fdd0eb792dee P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:04.403463 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3cb85eb3aa494f21bc8e506004318650" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:04.403630 16986 raft_consensus.cc:3060] T 3cb85eb3aa494f21bc8e506004318650 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.404318 16984 raft_consensus.cc:2468] T 37e07f15e43c49a4bf5ab1334dd85d22 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 1.
I20260929 02:01:04.404688 16985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "37e07f15e43c49a4bf5ab1334dd85d22" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:04.404917 16985 raft_consensus.cc:2376] T 37e07f15e43c49a4bf5ab1334dd85d22 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader pre-election vote request: Already granted yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 1. Re-sending same reply.
I20260929 02:01:04.404979 17189 leader_election.cc:304] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:04.405782 17348 raft_consensus.cc:2804] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:04.405949 17348 raft_consensus.cc:697] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Becoming Leader. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Running, Role: LEADER
I20260929 02:01:04.406266 17348 consensus_queue.cc:237] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.408504 17399 raft_consensus.cc:515] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.409149 17399 leader_election.cc:290] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:04.409746 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d9e051c702b42bc9392c5dcae42265d" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:04.409965 17251 raft_consensus.cc:3060] T 4d9e051c702b42bc9392c5dcae42265d P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.409961 16985 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c64e85b42e0d4618885975ac4125d4f2" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:04.410791 17056 leader_election.cc:304] T c64e85b42e0d4618885975ac4125d4f2 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc; no voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88
I20260929 02:01:04.411358 17436 raft_consensus.cc:3060] T c64e85b42e0d4618885975ac4125d4f2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.411504 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4d9e051c702b42bc9392c5dcae42265d" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:04.411667 16984 raft_consensus.cc:3060] T 4d9e051c702b42bc9392c5dcae42265d P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.411779 17398 raft_consensus.cc:697] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Becoming Leader. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Running, Role: LEADER
I20260929 02:01:04.412057 17398 consensus_queue.cc:237] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.413918 17428 consensus_queue.cc:1048] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:04.413990 17251 raft_consensus.cc:2468] T 4d9e051c702b42bc9392c5dcae42265d P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:04.414565 17055 leader_election.cc:304] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:04.414595 16984 raft_consensus.cc:2468] T 4d9e051c702b42bc9392c5dcae42265d P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:04.415508 17399 raft_consensus.cc:2804] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:04.415697 17399 raft_consensus.cc:697] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Becoming Leader. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Running, Role: LEADER
I20260929 02:01:04.415970 17399 consensus_queue.cc:237] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.421586 16986 raft_consensus.cc:2468] T 3cb85eb3aa494f21bc8e506004318650 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:04.422896 17436 raft_consensus.cc:2749] T c64e85b42e0d4618885975ac4125d4f2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:04.426878 16818 catalog_manager.cc:5838] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88 reported cstate change: term changed from 0 to 1, leader changed from <none> to d103dd4823044faebe46100a0dfbee88 (127.0.71.67). New cstate: current_term: 1 leader_uuid: "d103dd4823044faebe46100a0dfbee88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: HEALTHY } } }
I20260929 02:01:04.427765 16819 catalog_manager.cc:5838] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66). New cstate: current_term: 1 leader_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:04.430114 17398 consensus_queue.cc:1048] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:04.433881 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.433998 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.434242 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.434336 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.434548 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.434655 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.434868 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.435002 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.435411 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.435714 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.435971 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.436561 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.436829 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.437060 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.437471 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.437786 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.438087 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.441368 17337 consensus_queue.cc:1048] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:04.441682 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.441516 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":170,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:04.442857 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.443307 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.443415 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.163731
I20260929 02:01:04.448247 17337 consensus_queue.cc:1048] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:04.448735 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.004s sys 0.001s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":158,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:04.451157 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.013s user 0.003s sys 0.001s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":177,"lbm_writes_lt_1ms":4,"mutex_wait_us":7036,"reinsert_count":0,"update_count":1}
I20260929 02:01:04.451220 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.013s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":12819}
I20260929 02:01:04.451891 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.014s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":13462}
I20260929 02:01:04.454627 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.144132
I20260929 02:01:04.454982 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.144132
I20260929 02:01:04.455296 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.144132
I20260929 02:01:04.455286 17338 consensus_queue.cc:1048] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:04.455621 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.021s user 0.004s sys 0.001s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":162,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:04.456070 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.456563 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.457198 16819 catalog_manager.cc:5838] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66). New cstate: current_term: 1 leader_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:04.468956 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.025s user 0.011s sys 0.000s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":7045,"cfile_init":6,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":294,"lbm_read_time_us":651,"lbm_reads_lt_1ms":31,"lbm_write_time_us":344,"lbm_writes_lt_1ms":9,"mutex_wait_us":7764,"peak_mem_usage":32768,"reinsert_count":0,"update_count":2}
I20260929 02:01:04.469244 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.014s user 0.010s sys 0.000s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5808,"cfile_init":6,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":479,"lbm_read_time_us":590,"lbm_reads_lt_1ms":31,"lbm_write_time_us":350,"lbm_writes_lt_1ms":9,"peak_mem_usage":32768,"reinsert_count":0,"update_count":2}
I20260929 02:01:04.472790 16817 catalog_manager.cc:5838] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88 reported cstate change: term changed from 0 to 1, leader changed from <none> to d103dd4823044faebe46100a0dfbee88 (127.0.71.67). New cstate: current_term: 1 leader_uuid: "d103dd4823044faebe46100a0dfbee88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: HEALTHY } } }
I20260929 02:01:04.479948 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.480319 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.480420 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.480619 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.480767 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.480891 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.481058 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.481344 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.482087 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.482402 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.482646 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.482787 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.483141 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.483170 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.486768 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.002s sys 0.000s Metrics: {"bytes_written":466,"delete_count":0,"lbm_write_time_us":174,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":2}
I20260929 02:01:04.487010 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.001s sys 0.001s Metrics: {"bytes_written":466,"delete_count":0,"lbm_write_time_us":183,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":2}
I20260929 02:01:04.487327 17359 consensus_queue.cc:1048] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:04.488847 17436 consensus_queue.cc:1048] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:04.495319 17398 consensus_queue.cc:1048] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:04.495824 17376 consensus_queue.cc:1048] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:04.528299 17376 consensus_queue.cc:1048] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:04.533730 17337 consensus_queue.cc:1048] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:04.645417 17369 raft_consensus.cc:493] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:04.645644 17369 raft_consensus.cc:515] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } }
I20260929 02:01:04.646409 17369 leader_election.cc:290] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:04.646992 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3686e82fe0f4b6790a9a7f290c06876" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:04.647081 17252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3686e82fe0f4b6790a9a7f290c06876" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:04.647261 16986 raft_consensus.cc:2468] T c3686e82fe0f4b6790a9a7f290c06876 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:04.647411 17252 raft_consensus.cc:2468] T c3686e82fe0f4b6790a9a7f290c06876 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:04.647835 17056 leader_election.cc:304] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:04.648190 17369 raft_consensus.cc:2804] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:04.648285 17369 raft_consensus.cc:493] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:04.648381 17369 raft_consensus.cc:3060] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.650395 17369 raft_consensus.cc:515] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } }
I20260929 02:01:04.650990 17369 leader_election.cc:290] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:04.651394 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3686e82fe0f4b6790a9a7f290c06876" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:04.651472 17252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c3686e82fe0f4b6790a9a7f290c06876" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:04.651619 16986 raft_consensus.cc:3060] T c3686e82fe0f4b6790a9a7f290c06876 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.651659 17252 raft_consensus.cc:3060] T c3686e82fe0f4b6790a9a7f290c06876 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:04.654647 17252 raft_consensus.cc:2468] T c3686e82fe0f4b6790a9a7f290c06876 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:04.654647 16986 raft_consensus.cc:2468] T c3686e82fe0f4b6790a9a7f290c06876 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:04.655148 17055 leader_election.cc:304] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:04.655436 17369 raft_consensus.cc:2804] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:04.655652 17369 raft_consensus.cc:697] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Becoming Leader. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Running, Role: LEADER
I20260929 02:01:04.656020 17369 consensus_queue.cc:237] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } }
I20260929 02:01:04.659880 17337 consensus_queue.cc:1048] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:04.659989 16818 catalog_manager.cc:5838] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66). New cstate: current_term: 1 leader_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: HEALTHY } } }
I20260929 02:01:04.673316 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.673671 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.674479 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.675341 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.675773 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.200686
I20260929 02:01:04.676110 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.200686
I20260929 02:01:04.676410 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.200686
I20260929 02:01:04.676755 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.676890 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.677120 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:04.678661 16818 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44708:
table { table_name: "table-0" } modify_external_catalogs: true
I20260929 02:01:04.678879 16818 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44708:
table { table_name: "table-0" } modify_external_catalogs: true
I20260929 02:01:04.679949 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.001s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":206,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:04.680434 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.002s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":193,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:04.680733 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.121941
I20260929 02:01:04.680979 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.681017 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.121941
I20260929 02:01:04.686990 17376 consensus_queue.cc:1048] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:04.690749 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:04.691015 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.691345 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.697433 17376 consensus_queue.cc:1048] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:04.700762 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.020s user 0.008s sys 0.002s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":7149,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":296,"lbm_read_time_us":476,"lbm_reads_lt_1ms":27,"lbm_write_time_us":314,"lbm_writes_lt_1ms":9,"peak_mem_usage":32768,"reinsert_count":0,"update_count":3}
I20260929 02:01:04.701381 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.701901 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.702208 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.702497 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.703078 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.703435 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.706068 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet c3686e82fe0f4b6790a9a7f290c06876
I20260929 02:01:04.707139 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 9103285f38f149a8ab80fdd0eb792dee
I20260929 02:01:04.707120 16966 tablet_service.cc:1558] Processing DeleteTablet for tablet c3686e82fe0f4b6790a9a7f290c06876 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:04.707192 17233 tablet_service.cc:1558] Processing DeleteTablet for tablet c3686e82fe0f4b6790a9a7f290c06876 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:04.708027 16955 tablet_service.cc:1558] Processing DeleteTablet for tablet 9103285f38f149a8ab80fdd0eb792dee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:04.708945 17218 tablet_service.cc:1558] Processing DeleteTablet for tablet 9103285f38f149a8ab80fdd0eb792dee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:04.709245 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 9a0e71f5f586410fabe38e3c3b26dfd0
I20260929 02:01:04.709893 16955 tablet_service.cc:1558] Processing DeleteTablet for tablet 9a0e71f5f586410fabe38e3c3b26dfd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:04.712200 17457 tablet_replica.cc:333] T c3686e82fe0f4b6790a9a7f290c06876 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:04.712714 17457 raft_consensus.cc:2243] T c3686e82fe0f4b6790a9a7f290c06876 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:04.713073 17457 raft_consensus.cc:2272] T c3686e82fe0f4b6790a9a7f290c06876 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:04.714632 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.013s user 0.003s sys 0.000s Metrics: {"bytes_written":937,"delete_count":0,"lbm_write_time_us":132,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":17}
I20260929 02:01:04.715097 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.124800
I20260929 02:01:04.715518 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.124800
I20260929 02:01:04.715735 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.717372 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.015s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":14564}
I20260929 02:01:04.717430 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.015s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":14594}
I20260929 02:01:04.717353 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.015s user 0.006s sys 0.000s Metrics: {"bytes_written":937,"delete_count":0,"lbm_write_time_us":223,"lbm_writes_lt_1ms":4,"mutex_wait_us":7783,"reinsert_count":0,"spinlock_wait_cycles":896,"update_count":17}
I20260929 02:01:04.718722 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.042s user 0.010s sys 0.000s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5912,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":5552,"dirs.run_cpu_time_us":333,"dirs.run_wall_time_us":7069,"lbm_read_time_us":514,"lbm_reads_lt_1ms":27,"lbm_write_time_us":326,"lbm_writes_lt_1ms":9,"mutex_wait_us":8405,"peak_mem_usage":32768,"reinsert_count":0,"update_count":3}
I20260929 02:01:04.719321 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.150281
I20260929 02:01:04.719632 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.150281
I20260929 02:01:04.719805 17233 tablet_service.cc:1558] Processing DeleteTablet for tablet 9a0e71f5f586410fabe38e3c3b26dfd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:04.719894 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.150281
I20260929 02:01:04.720171 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.150281
I20260929 02:01:04.720587 17456 tablet_replica.cc:333] T c3686e82fe0f4b6790a9a7f290c06876 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:04.721053 17456 raft_consensus.cc:2243] T c3686e82fe0f4b6790a9a7f290c06876 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:04.721352 17456 raft_consensus.cc:2272] T c3686e82fe0f4b6790a9a7f290c06876 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:04.722728 17456 ts_tablet_manager.cc:1962] T c3686e82fe0f4b6790a9a7f290c06876 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:04.724751 17086 tablet_service.cc:1558] Processing DeleteTablet for tablet 2bc3dbd1c82e4d2cb9577c51d9b19224 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:04.724933 17097 tablet_service.cc:1558] Processing DeleteTablet for tablet 9a0e71f5f586410fabe38e3c3b26dfd0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:04.724792 17100 tablet_service.cc:1558] Processing DeleteTablet for tablet 9103285f38f149a8ab80fdd0eb792dee with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:04.725369 17092 tablet_service.cc:1558] Processing DeleteTablet for tablet c3686e82fe0f4b6790a9a7f290c06876 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:04.725669 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.725932 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.730118 17456 ts_tablet_manager.cc:1975] T c3686e82fe0f4b6790a9a7f290c06876 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260929 02:01:04.730268 17456 log.cc:1199] T c3686e82fe0f4b6790a9a7f290c06876 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/c3686e82fe0f4b6790a9a7f290c06876
I20260929 02:01:04.730710 17458 tablet_replica.cc:333] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:04.730731 17456 ts_tablet_manager.cc:1996] T c3686e82fe0f4b6790a9a7f290c06876 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:04.731289 17458 raft_consensus.cc:2243] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:04.731607 17458 raft_consensus.cc:2272] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:04.734366 17337 consensus_queue.cc:1048] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:04.735699 17233 tablet_service.cc:1558] Processing DeleteTablet for tablet 2bc3dbd1c82e4d2cb9577c51d9b19224 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:04.736733 17456 tablet_replica.cc:333] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:04.736981 17456 raft_consensus.cc:2243] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:04.737212 17456 raft_consensus.cc:2272] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:04.737506 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet c3686e82fe0f4b6790a9a7f290c06876 (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:04.738117 17456 ts_tablet_manager.cc:1962] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:04.741789 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet ca4fa192c95745bfa163ef63f98d29a5
I20260929 02:01:04.743031 17233 tablet_service.cc:1558] Processing DeleteTablet for tablet ca4fa192c95745bfa163ef63f98d29a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:04.743248 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 7a962f60729242a1bcebe88c89f8f895
I20260929 02:01:04.743520 16966 tablet_service.cc:1558] Processing DeleteTablet for tablet ca4fa192c95745bfa163ef63f98d29a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:04.744185 16966 tablet_service.cc:1558] Processing DeleteTablet for tablet 7a962f60729242a1bcebe88c89f8f895 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:04.744297 17233 tablet_service.cc:1558] Processing DeleteTablet for tablet 7a962f60729242a1bcebe88c89f8f895 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:04.744514 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet b2d556cada174f83a633a7a8836826ce
I20260929 02:01:04.744940 17086 tablet_service.cc:1558] Processing DeleteTablet for tablet ca4fa192c95745bfa163ef63f98d29a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:04.745177 17092 tablet_service.cc:1558] Processing DeleteTablet for tablet 7a962f60729242a1bcebe88c89f8f895 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:04.745404 17086 tablet_service.cc:1558] Processing DeleteTablet for tablet b2d556cada174f83a633a7a8836826ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:04.745674 17233 tablet_service.cc:1558] Processing DeleteTablet for tablet b2d556cada174f83a633a7a8836826ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:04.745843 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet b591026c764b48a9af227b7be6dd0ee6
I20260929 02:01:04.746657 17086 tablet_service.cc:1558] Processing DeleteTablet for tablet b591026c764b48a9af227b7be6dd0ee6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:04.746831 17456 ts_tablet_manager.cc:1975] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:04.747000 17456 log.cc:1199] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/9103285f38f149a8ab80fdd0eb792dee
I20260929 02:01:04.747203 17233 tablet_service.cc:1558] Processing DeleteTablet for tablet b591026c764b48a9af227b7be6dd0ee6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:04.747339 16955 tablet_service.cc:1558] Processing DeleteTablet for tablet b591026c764b48a9af227b7be6dd0ee6 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:04.747529 17456 ts_tablet_manager.cc:1996] T 9103285f38f149a8ab80fdd0eb792dee P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:04.747658 16966 tablet_service.cc:1558] Processing DeleteTablet for tablet b2d556cada174f83a633a7a8836826ce with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:04.747763 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 8601b0ded51b49d0abeb95f7e5fedf8c
I20260929 02:01:04.748482 17456 tablet_replica.cc:333] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:04.748694 17456 raft_consensus.cc:2243] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:04.748889 17456 raft_consensus.cc:2272] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:04.749480 17233 tablet_service.cc:1558] Processing DeleteTablet for tablet 8601b0ded51b49d0abeb95f7e5fedf8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:04.749609 17086 tablet_service.cc:1558] Processing DeleteTablet for tablet 8601b0ded51b49d0abeb95f7e5fedf8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:04.749644 17456 ts_tablet_manager.cc:1962] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:04.749701 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 3cb85eb3aa494f21bc8e506004318650
I20260929 02:01:04.749763 17457 ts_tablet_manager.cc:1962] T c3686e82fe0f4b6790a9a7f290c06876 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:04.750272 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 9103285f38f149a8ab80fdd0eb792dee (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:04.750998 17086 tablet_service.cc:1558] Processing DeleteTablet for tablet 3cb85eb3aa494f21bc8e506004318650 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:04.751197 17233 tablet_service.cc:1558] Processing DeleteTablet for tablet 3cb85eb3aa494f21bc8e506004318650 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:04.751438 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet c7189d73aa7c43b99293f2da0e47338b
I20260929 02:01:04.751652 16955 tablet_service.cc:1558] Processing DeleteTablet for tablet 3cb85eb3aa494f21bc8e506004318650 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:04.752142 16966 tablet_service.cc:1558] Processing DeleteTablet for tablet 8601b0ded51b49d0abeb95f7e5fedf8c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:04.752753 17086 tablet_service.cc:1558] Processing DeleteTablet for tablet c7189d73aa7c43b99293f2da0e47338b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:04.753058 17233 tablet_service.cc:1558] Processing DeleteTablet for tablet c7189d73aa7c43b99293f2da0e47338b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:04.753167 16966 tablet_service.cc:1558] Processing DeleteTablet for tablet c7189d73aa7c43b99293f2da0e47338b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:04.753278 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet c64e85b42e0d4618885975ac4125d4f2
I20260929 02:01:04.754263 16966 tablet_service.cc:1558] Processing DeleteTablet for tablet c64e85b42e0d4618885975ac4125d4f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:04.754703 17086 tablet_service.cc:1558] Processing DeleteTablet for tablet c64e85b42e0d4618885975ac4125d4f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:04.755209 17233 tablet_service.cc:1558] Processing DeleteTablet for tablet c64e85b42e0d4618885975ac4125d4f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:04.755525 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 0aa899b966dd481088d18119ee57d81a
I20260929 02:01:04.756278 16966 tablet_service.cc:1558] Processing DeleteTablet for tablet 0aa899b966dd481088d18119ee57d81a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:04.756724 17456 ts_tablet_manager.cc:1975] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:04.756793 17086 tablet_service.cc:1558] Processing DeleteTablet for tablet 0aa899b966dd481088d18119ee57d81a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:04.756850 17457 ts_tablet_manager.cc:1975] T c3686e82fe0f4b6790a9a7f290c06876 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260929 02:01:04.756999 17457 log.cc:1199] T c3686e82fe0f4b6790a9a7f290c06876 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/c3686e82fe0f4b6790a9a7f290c06876
I20260929 02:01:04.757121 17456 log.cc:1199] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/9a0e71f5f586410fabe38e3c3b26dfd0
I20260929 02:01:04.757315 17233 tablet_service.cc:1558] Processing DeleteTablet for tablet 0aa899b966dd481088d18119ee57d81a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:04.757529 17457 ts_tablet_manager.cc:1996] T c3686e82fe0f4b6790a9a7f290c06876 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:04.757544 17456 ts_tablet_manager.cc:1996] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:04.757537 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 43f983a2802b4ad3871d743b42b5deea
I20260929 02:01:04.758227 16966 tablet_service.cc:1558] Processing DeleteTablet for tablet 43f983a2802b4ad3871d743b42b5deea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:04.758492 17456 tablet_replica.cc:333] T ca4fa192c95745bfa163ef63f98d29a5 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:04.758689 17456 raft_consensus.cc:2243] T ca4fa192c95745bfa163ef63f98d29a5 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:04.758706 17086 tablet_service.cc:1558] Processing DeleteTablet for tablet 43f983a2802b4ad3871d743b42b5deea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:04.758864 17456 raft_consensus.cc:2272] T ca4fa192c95745bfa163ef63f98d29a5 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:04.759181 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 9a0e71f5f586410fabe38e3c3b26dfd0 (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:04.759181 17233 tablet_service.cc:1558] Processing DeleteTablet for tablet 43f983a2802b4ad3871d743b42b5deea with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:04.759467 17457 tablet_replica.cc:333] T 9103285f38f149a8ab80fdd0eb792dee P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:04.759466 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet c3686e82fe0f4b6790a9a7f290c06876 (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:04.759675 17456 ts_tablet_manager.cc:1962] T ca4fa192c95745bfa163ef63f98d29a5 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:04.759681 17457 raft_consensus.cc:2243] T 9103285f38f149a8ab80fdd0eb792dee P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:04.759857 17457 raft_consensus.cc:2272] T 9103285f38f149a8ab80fdd0eb792dee P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:04.759707 17458 ts_tablet_manager.cc:1962] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:04.761108 17457 ts_tablet_manager.cc:1962] T 9103285f38f149a8ab80fdd0eb792dee P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:04.766489 17456 ts_tablet_manager.cc:1975] T ca4fa192c95745bfa163ef63f98d29a5 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:04.766638 17456 log.cc:1199] T ca4fa192c95745bfa163ef63f98d29a5 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/ca4fa192c95745bfa163ef63f98d29a5
I20260929 02:01:04.767033 17456 ts_tablet_manager.cc:1996] T ca4fa192c95745bfa163ef63f98d29a5 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:04.767277 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.052s user 0.009s sys 0.002s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":7396,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":85,"lbm_read_time_us":484,"lbm_reads_lt_1ms":25,"lbm_write_time_us":365,"lbm_writes_lt_1ms":9,"peak_mem_usage":32768,"reinsert_count":0,"update_count":17}
I20260929 02:01:04.767905 17457 ts_tablet_manager.cc:1975] T 9103285f38f149a8ab80fdd0eb792dee P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:04.768045 17457 log.cc:1199] T 9103285f38f149a8ab80fdd0eb792dee P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/9103285f38f149a8ab80fdd0eb792dee
I20260929 02:01:04.768348 17456 tablet_replica.cc:333] T 7a962f60729242a1bcebe88c89f8f895 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:04.768317 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.022s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.768471 17457 ts_tablet_manager.cc:1996] T 9103285f38f149a8ab80fdd0eb792dee P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:04.768508 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet ca4fa192c95745bfa163ef63f98d29a5 (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:04.768615 17456 raft_consensus.cc:2243] T 7a962f60729242a1bcebe88c89f8f895 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:04.768826 17456 raft_consensus.cc:2272] T 7a962f60729242a1bcebe88c89f8f895 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:04.769120 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.049s user 0.005s sys 0.006s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":6159,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":2058,"lbm_read_time_us":549,"lbm_reads_lt_1ms":25,"lbm_write_time_us":335,"lbm_writes_lt_1ms":9,"peak_mem_usage":32768,"reinsert_count":0,"update_count":17}
I20260929 02:01:04.769474 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 9103285f38f149a8ab80fdd0eb792dee (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:04.769745 17456 ts_tablet_manager.cc:1962] T 7a962f60729242a1bcebe88c89f8f895 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:04.769991 17458 ts_tablet_manager.cc:1975] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:04.769894 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 4d9e051c702b42bc9392c5dcae42265d
I20260929 02:01:04.770254 17458 log.cc:1199] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/2bc3dbd1c82e4d2cb9577c51d9b19224
I20260929 02:01:04.770843 16966 tablet_service.cc:1558] Processing DeleteTablet for tablet 4d9e051c702b42bc9392c5dcae42265d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:04.771196 17458 ts_tablet_manager.cc:1996] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:04.771426 17086 tablet_service.cc:1558] Processing DeleteTablet for tablet 4d9e051c702b42bc9392c5dcae42265d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:04.771909 17233 tablet_service.cc:1558] Processing DeleteTablet for tablet 4d9e051c702b42bc9392c5dcae42265d with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:04.772101 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 2bc3dbd1c82e4d2cb9577c51d9b19224
I20260929 02:01:04.772764 16966 tablet_service.cc:1558] Processing DeleteTablet for tablet 2bc3dbd1c82e4d2cb9577c51d9b19224 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:04.773303 17086 tablet_service.cc:1558] Processing DeleteTablet for tablet 2bc3dbd1c82e4d2cb9577c51d9b19224 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:04.773772 17233 tablet_service.cc:1558] Processing DeleteTablet for tablet 2bc3dbd1c82e4d2cb9577c51d9b19224 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:04.773988 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 37e07f15e43c49a4bf5ab1334dd85d22
I20260929 02:01:04.774430 17458 tablet_replica.cc:333] T 9103285f38f149a8ab80fdd0eb792dee P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:04.774649 16966 tablet_service.cc:1558] Processing DeleteTablet for tablet 37e07f15e43c49a4bf5ab1334dd85d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:04.775209 17233 tablet_service.cc:1558] Processing DeleteTablet for tablet 37e07f15e43c49a4bf5ab1334dd85d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:04.775234 17086 tablet_service.cc:1558] Processing DeleteTablet for tablet 37e07f15e43c49a4bf5ab1334dd85d22 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:04 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:04.776129 17458 raft_consensus.cc:2243] T 9103285f38f149a8ab80fdd0eb792dee P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:04.776216 17456 ts_tablet_manager.cc:1975] T 7a962f60729242a1bcebe88c89f8f895 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:04.776338 17456 log.cc:1199] T 7a962f60729242a1bcebe88c89f8f895 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/7a962f60729242a1bcebe88c89f8f895
I20260929 02:01:04.776482 17458 raft_consensus.cc:2272] T 9103285f38f149a8ab80fdd0eb792dee P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:04.776980 17456 ts_tablet_manager.cc:1996] T 7a962f60729242a1bcebe88c89f8f895 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:04.777307 16818 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44708:
name: "table-1"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "col"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 16
seed: 0
}
}
I20260929 02:01:04.777892 17456 tablet_replica.cc:333] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:04.778095 17456 raft_consensus.cc:2243] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:04.778404 17456 raft_consensus.cc:2272] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:04.778422 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 7a962f60729242a1bcebe88c89f8f895 (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:04.778690 17458 ts_tablet_manager.cc:1962] T 9103285f38f149a8ab80fdd0eb792dee P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:04.779369 17457 tablet_replica.cc:333] T 9a0e71f5f586410fabe38e3c3b26dfd0 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:04.779573 17457 raft_consensus.cc:2243] T 9a0e71f5f586410fabe38e3c3b26dfd0 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:04.779655 17456 ts_tablet_manager.cc:1962] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:04.779778 17457 raft_consensus.cc:2272] T 9a0e71f5f586410fabe38e3c3b26dfd0 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:04.780791 17457 ts_tablet_manager.cc:1962] T 9a0e71f5f586410fabe38e3c3b26dfd0 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:04.785333 17458 ts_tablet_manager.cc:1975] T 9103285f38f149a8ab80fdd0eb792dee P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:04.785504 17458 log.cc:1199] T 9103285f38f149a8ab80fdd0eb792dee P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/9103285f38f149a8ab80fdd0eb792dee
I20260929 02:01:04.785908 17458 ts_tablet_manager.cc:1996] T 9103285f38f149a8ab80fdd0eb792dee P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:04.786840 17456 ts_tablet_manager.cc:1975] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:04.786991 17456 log.cc:1199] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/b591026c764b48a9af227b7be6dd0ee6
I20260929 02:01:04.787047 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 9103285f38f149a8ab80fdd0eb792dee (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:04.787407 17456 ts_tablet_manager.cc:1996] T b591026c764b48a9af227b7be6dd0ee6 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
W20260929 02:01:04.787592 16818 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table-1 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:01:04.787909 17457 ts_tablet_manager.cc:1975] T 9a0e71f5f586410fabe38e3c3b26dfd0 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:04.788089 17457 log.cc:1199] T 9a0e71f5f586410fabe38e3c3b26dfd0 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/9a0e71f5f586410fabe38e3c3b26dfd0
I20260929 02:01:04.788352 17456 tablet_replica.cc:333] T b2d556cada174f83a633a7a8836826ce P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:04.788517 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet b591026c764b48a9af227b7be6dd0ee6 (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:04.788600 17456 raft_consensus.cc:2243] T b2d556cada174f83a633a7a8836826ce P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:04.788826 17456 raft_consensus.cc:2272] T b2d556cada174f83a633a7a8836826ce P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:04.789165 17457 ts_tablet_manager.cc:1996] T 9a0e71f5f586410fabe38e3c3b26dfd0 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:04.789657 17456 ts_tablet_manager.cc:1962] T b2d556cada174f83a633a7a8836826ce P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:04.790282 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 9a0e71f5f586410fabe38e3c3b26dfd0 (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:04.790621 17457 tablet_replica.cc:333] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:04.790853 17457 raft_consensus.cc:2243] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:04.791061 17457 raft_consensus.cc:2272] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:04.791985 17457 ts_tablet_manager.cc:1962] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 02:01:04.795397 17055 consensus_peers.cc:597] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc -> Peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Couldn't send request to peer d103dd4823044faebe46100a0dfbee88. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 9a0e71f5f586410fabe38e3c3b26dfd0. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:01:04.796178 17456 ts_tablet_manager.cc:1975] T b2d556cada174f83a633a7a8836826ce P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:04.796305 17456 log.cc:1199] T b2d556cada174f83a633a7a8836826ce P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/b2d556cada174f83a633a7a8836826ce
I20260929 02:01:04.796805 17456 ts_tablet_manager.cc:1996] T b2d556cada174f83a633a7a8836826ce P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:04.797739 17456 tablet_replica.cc:333] T 3cb85eb3aa494f21bc8e506004318650 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:04.797904 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet b2d556cada174f83a633a7a8836826ce (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:04.797942 17456 raft_consensus.cc:2243] T 3cb85eb3aa494f21bc8e506004318650 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:04.798094 17456 raft_consensus.cc:2272] T 3cb85eb3aa494f21bc8e506004318650 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:04.798646 17456 ts_tablet_manager.cc:1962] T 3cb85eb3aa494f21bc8e506004318650 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:04.799536 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.800159 17457 ts_tablet_manager.cc:1975] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:04.800362 17457 log.cc:1199] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/2bc3dbd1c82e4d2cb9577c51d9b19224
I20260929 02:01:04.800798 17457 ts_tablet_manager.cc:1996] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:04.801653 17457 tablet_replica.cc:333] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:04.801803 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 2bc3dbd1c82e4d2cb9577c51d9b19224 (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:04.801864 17457 raft_consensus.cc:2243] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:04.802392 17457 raft_consensus.cc:2272] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:04.803061 17457 ts_tablet_manager.cc:1962] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:04.805100 17456 ts_tablet_manager.cc:1975] T 3cb85eb3aa494f21bc8e506004318650 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260929 02:01:04.805239 17456 log.cc:1199] T 3cb85eb3aa494f21bc8e506004318650 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/3cb85eb3aa494f21bc8e506004318650
I20260929 02:01:04.805627 17456 ts_tablet_manager.cc:1996] T 3cb85eb3aa494f21bc8e506004318650 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:04.806548 17456 tablet_replica.cc:333] T 8601b0ded51b49d0abeb95f7e5fedf8c P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:04.806725 17456 raft_consensus.cc:2243] T 8601b0ded51b49d0abeb95f7e5fedf8c P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:04.806697 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 3cb85eb3aa494f21bc8e506004318650 (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:04.806895 17456 raft_consensus.cc:2272] T 8601b0ded51b49d0abeb95f7e5fedf8c P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:04.807687 17456 ts_tablet_manager.cc:1962] T 8601b0ded51b49d0abeb95f7e5fedf8c P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:04.807894 17457 ts_tablet_manager.cc:1975] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:04.808033 17457 log.cc:1199] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/ca4fa192c95745bfa163ef63f98d29a5
I20260929 02:01:04.808512 17457 ts_tablet_manager.cc:1996] T ca4fa192c95745bfa163ef63f98d29a5 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:04.809633 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet ca4fa192c95745bfa163ef63f98d29a5 (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:04.810014 17457 tablet_replica.cc:333] T 7a962f60729242a1bcebe88c89f8f895 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:04.810252 17457 raft_consensus.cc:2243] T 7a962f60729242a1bcebe88c89f8f895 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:04.810431 17457 raft_consensus.cc:2272] T 7a962f60729242a1bcebe88c89f8f895 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:04.811112 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.811386 17457 ts_tablet_manager.cc:1962] T 7a962f60729242a1bcebe88c89f8f895 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:04.816740 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.017s user 0.012s sys 0.001s Metrics: {"bytes_written":6939,"cfile_init":1,"dirs.queue_time_us":54,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":694,"drs_written":1,"lbm_read_time_us":80,"lbm_reads_lt_1ms":4,"lbm_write_time_us":832,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":17}
I20260929 02:01:04.817723 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.292615
I20260929 02:01:04.818070 17457 ts_tablet_manager.cc:1975] T 7a962f60729242a1bcebe88c89f8f895 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:04.818782 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.292615
I20260929 02:01:04.819142 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260929 02:01:04.819442 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260929 02:01:04.819862 17458 tablet_replica.cc:333] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:04.820180 17458 raft_consensus.cc:2243] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:04.820925 17458 raft_consensus.cc:2272] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:04.821976 17458 ts_tablet_manager.cc:1962] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:04.824992 17456 ts_tablet_manager.cc:1975] T 8601b0ded51b49d0abeb95f7e5fedf8c P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:04.825157 17456 log.cc:1199] T 8601b0ded51b49d0abeb95f7e5fedf8c P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/8601b0ded51b49d0abeb95f7e5fedf8c
I20260929 02:01:04.825700 17456 ts_tablet_manager.cc:1996] T 8601b0ded51b49d0abeb95f7e5fedf8c P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
W20260929 02:01:04.826259 17189 consensus_peers.cc:597] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88 -> Peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Couldn't send request to peer 97df55a0361841319adfae27c9787b2f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:01:04.826694 17456 tablet_replica.cc:333] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:04.826900 17456 raft_consensus.cc:2243] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:04.827219 17456 raft_consensus.cc:2272] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:04.828907 17456 ts_tablet_manager.cc:1962] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:04.828981 17457 log.cc:1199] T 7a962f60729242a1bcebe88c89f8f895 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/7a962f60729242a1bcebe88c89f8f895
I20260929 02:01:04.829092 17458 ts_tablet_manager.cc:1975] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:04.829221 17458 log.cc:1199] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/9a0e71f5f586410fabe38e3c3b26dfd0
I20260929 02:01:04.829638 17458 ts_tablet_manager.cc:1996] T 9a0e71f5f586410fabe38e3c3b26dfd0 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:04.830700 17457 ts_tablet_manager.cc:1996] T 7a962f60729242a1bcebe88c89f8f895 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:04.830855 17458 tablet_replica.cc:333] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:04.831072 17458 raft_consensus.cc:2243] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:04.831303 17458 pending_rounds.cc:70] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc: Trying to abort 1 pending ops.
I20260929 02:01:04.831488 17458 pending_rounds.cc:77] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7334491196033851392 op_type: NO_OP noop_request { }
I20260929 02:01:04.831640 17458 raft_consensus.cc:2889] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260929 02:01:04.831794 17458 raft_consensus.cc:2272] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:04.831933 17457 tablet_replica.cc:333] T b2d556cada174f83a633a7a8836826ce P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:04.832134 17457 raft_consensus.cc:2243] T b2d556cada174f83a633a7a8836826ce P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:04.832329 17457 raft_consensus.cc:2272] T b2d556cada174f83a633a7a8836826ce P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:04.832716 17458 ts_tablet_manager.cc:1962] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:04.833138 17457 ts_tablet_manager.cc:1962] T b2d556cada174f83a633a7a8836826ce P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:04.834731 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.023s user 0.008s sys 0.004s Metrics: {"bytes_written":6939,"cfile_init":1,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":136,"dirs.run_wall_time_us":1154,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":957,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":17}
I20260929 02:01:04.835786 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260929 02:01:04.836122 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260929 02:01:04.836414 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260929 02:01:04.836699 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.873843
I20260929 02:01:04.837015 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20260929 02:01:04.837625 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.837838 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 7a962f60729242a1bcebe88c89f8f895 (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:04.838030 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 8601b0ded51b49d0abeb95f7e5fedf8c (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:04.839064 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 9a0e71f5f586410fabe38e3c3b26dfd0 (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:04.839540 17458 ts_tablet_manager.cc:1975] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:04.839653 17458 log.cc:1199] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/c3686e82fe0f4b6790a9a7f290c06876
I20260929 02:01:04.840056 17458 ts_tablet_manager.cc:1996] T c3686e82fe0f4b6790a9a7f290c06876 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:04.841413 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet c3686e82fe0f4b6790a9a7f290c06876 (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:04.843707 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.845419 17456 ts_tablet_manager.cc:1975] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:04.845696 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.845932 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.846084 17456 log.cc:1199] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/c7189d73aa7c43b99293f2da0e47338b
I20260929 02:01:04.846305 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.846355 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.846616 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.846899 17456 ts_tablet_manager.cc:1996] T c7189d73aa7c43b99293f2da0e47338b P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:04.846951 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.848102 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet c7189d73aa7c43b99293f2da0e47338b (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:04.848183 17456 tablet_replica.cc:333] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:04.848492 17456 raft_consensus.cc:2243] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:04.848918 17456 raft_consensus.cc:2272] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:04.849133 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.849378 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.850037 17456 ts_tablet_manager.cc:1962] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:04.850096 17457 ts_tablet_manager.cc:1975] T b2d556cada174f83a633a7a8836826ce P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:04.850234 17457 log.cc:1199] T b2d556cada174f83a633a7a8836826ce P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/b2d556cada174f83a633a7a8836826ce
I20260929 02:01:04.850639 17457 ts_tablet_manager.cc:1996] T b2d556cada174f83a633a7a8836826ce P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:04.851752 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet b2d556cada174f83a633a7a8836826ce (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:04.852164 17457 tablet_replica.cc:333] T b591026c764b48a9af227b7be6dd0ee6 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:04.852551 17457 raft_consensus.cc:2243] T b591026c764b48a9af227b7be6dd0ee6 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:04.853123 17457 raft_consensus.cc:2272] T b591026c764b48a9af227b7be6dd0ee6 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:04.853804 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.003s sys 0.001s Metrics: {"bytes_written":683,"delete_count":0,"lbm_write_time_us":161,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":16}
I20260929 02:01:04.854272 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:04.857488 16966 tablet_service.cc:1511] Processing CreateTablet for tablet ea01c17d8d4b405ab885e7eec91b5cbc (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.857942 16966 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea01c17d8d4b405ab885e7eec91b5cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.858738 17092 tablet_service.cc:1511] Processing CreateTablet for tablet 20a08b5cc6ae46ebb4696c9f3c83c5fa (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.859182 17092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20a08b5cc6ae46ebb4696c9f3c83c5fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.860847 16955 tablet_service.cc:1511] Processing CreateTablet for tablet 20a08b5cc6ae46ebb4696c9f3c83c5fa (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.861179 16955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20a08b5cc6ae46ebb4696c9f3c83c5fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.864849 17086 tablet_service.cc:1511] Processing CreateTablet for tablet ea01c17d8d4b405ab885e7eec91b5cbc (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.865285 17086 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea01c17d8d4b405ab885e7eec91b5cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.866358 16951 tablet_service.cc:1511] Processing CreateTablet for tablet 2e9a65b8f4be4f1d99eef1517d20743c (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.866605 16954 tablet_service.cc:1511] Processing CreateTablet for tablet 2486911d1a704e159ed5ad88ebdd82c1 (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.866711 16951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e9a65b8f4be4f1d99eef1517d20743c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.867018 16954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2486911d1a704e159ed5ad88ebdd82c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.867110 16953 tablet_service.cc:1511] Processing CreateTablet for tablet 7822796e1a01411aaa1c2b394cc27263 (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.867442 16953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7822796e1a01411aaa1c2b394cc27263. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.868813 16952 tablet_service.cc:1511] Processing CreateTablet for tablet 9603392111a04ed395f159d69ac4801c (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.869182 16952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9603392111a04ed395f159d69ac4801c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.870935 16956 tablet_service.cc:1511] Processing CreateTablet for tablet 8cd30e2805234d9fadc0d2ad839a6bc3 (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.871333 16956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cd30e2805234d9fadc0d2ad839a6bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.871579 17458 tablet_replica.cc:333] T ca4fa192c95745bfa163ef63f98d29a5 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:04.871819 17458 raft_consensus.cc:2243] T ca4fa192c95745bfa163ef63f98d29a5 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:04.872020 17458 raft_consensus.cc:2272] T ca4fa192c95745bfa163ef63f98d29a5 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:04.872783 16960 tablet_service.cc:1511] Processing CreateTablet for tablet 3f0398c024bf4cc48ec1670fc847c7ab (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.873106 16960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f0398c024bf4cc48ec1670fc847c7ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.875166 17097 tablet_service.cc:1511] Processing CreateTablet for tablet 9603392111a04ed395f159d69ac4801c (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.875828 17100 tablet_service.cc:1511] Processing CreateTablet for tablet 7822796e1a01411aaa1c2b394cc27263 (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.876353 17090 tablet_service.cc:1511] Processing CreateTablet for tablet 2486911d1a704e159ed5ad88ebdd82c1 (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.877049 17085 tablet_service.cc:1511] Processing CreateTablet for tablet 8cd30e2805234d9fadc0d2ad839a6bc3 (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.877571 17087 tablet_service.cc:1511] Processing CreateTablet for tablet 3f0398c024bf4cc48ec1670fc847c7ab (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.878159 17089 tablet_service.cc:1511] Processing CreateTablet for tablet 2e9a65b8f4be4f1d99eef1517d20743c (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.878697 17099 tablet_service.cc:1511] Processing CreateTablet for tablet 00f69d45340f4d578a7f3b810c6f9fec (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.879552 16965 tablet_service.cc:1511] Processing CreateTablet for tablet 00f69d45340f4d578a7f3b810c6f9fec (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.880064 16965 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00f69d45340f4d578a7f3b810c6f9fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.879981 16961 tablet_service.cc:1511] Processing CreateTablet for tablet a4083e6b1e594ee88efbd7e1f04f70e0 (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.880309 17458 ts_tablet_manager.cc:1962] T ca4fa192c95745bfa163ef63f98d29a5 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:04.880347 16961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4083e6b1e594ee88efbd7e1f04f70e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.880695 16964 tablet_service.cc:1511] Processing CreateTablet for tablet 85b63af0846742c99534e9bd449181d3 (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.881167 16964 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85b63af0846742c99534e9bd449181d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.885221 16962 tablet_service.cc:1511] Processing CreateTablet for tablet d5da16034f2244a2b335f89f6ab7caa9 (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.885605 16962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5da16034f2244a2b335f89f6ab7caa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.886504 16963 tablet_service.cc:1511] Processing CreateTablet for tablet 5e7d7ccf81494fec9e066824c69abd61 (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.886879 16963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e7d7ccf81494fec9e066824c69abd61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.890414 17088 tablet_service.cc:1511] Processing CreateTablet for tablet a4083e6b1e594ee88efbd7e1f04f70e0 (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.890992 17095 tablet_service.cc:1511] Processing CreateTablet for tablet d5da16034f2244a2b335f89f6ab7caa9 (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.891703 17093 tablet_service.cc:1511] Processing CreateTablet for tablet 85b63af0846742c99534e9bd449181d3 (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.895224 16804 tablet.cc:2133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260929 02:01:04.896378 17094 tablet_service.cc:1511] Processing CreateTablet for tablet 5e7d7ccf81494fec9e066824c69abd61 (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.897799 16963 tablet_service.cc:1511] Processing CreateTablet for tablet 044313d0ac094292808355c152d310aa (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.898228 16963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 044313d0ac094292808355c152d310aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.900049 17097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9603392111a04ed395f159d69ac4801c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.900256 17099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00f69d45340f4d578a7f3b810c6f9fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.900537 17095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5da16034f2244a2b335f89f6ab7caa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.900722 17090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2486911d1a704e159ed5ad88ebdd82c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.900935 17093 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85b63af0846742c99534e9bd449181d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.901114 17089 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e9a65b8f4be4f1d99eef1517d20743c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.901553 17100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7822796e1a01411aaa1c2b394cc27263. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.902006 17088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4083e6b1e594ee88efbd7e1f04f70e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.902405 17085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cd30e2805234d9fadc0d2ad839a6bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.902796 17087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f0398c024bf4cc48ec1670fc847c7ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.903411 17094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e7d7ccf81494fec9e066824c69abd61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.907776 16963 tablet_service.cc:1511] Processing CreateTablet for tablet 7f3e8d6962a940868e1ab2caa707cff1 (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.908200 16963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f3e8d6962a940868e1ab2caa707cff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.913946 17458 ts_tablet_manager.cc:1975] T ca4fa192c95745bfa163ef63f98d29a5 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:04.914213 17458 log.cc:1199] T ca4fa192c95745bfa163ef63f98d29a5 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/ca4fa192c95745bfa163ef63f98d29a5
I20260929 02:01:04.915835 17458 ts_tablet_manager.cc:1996] T ca4fa192c95745bfa163ef63f98d29a5 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:04.918061 17458 tablet_replica.cc:333] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:04.918270 17458 raft_consensus.cc:2243] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:04.918637 17458 raft_consensus.cc:2272] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:04.920181 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet ca4fa192c95745bfa163ef63f98d29a5 (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:04.921201 17458 ts_tablet_manager.cc:1962] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:04.921291 17096 tablet_service.cc:1511] Processing CreateTablet for tablet 044313d0ac094292808355c152d310aa (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.921365 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.073s user 0.003s sys 0.002s Metrics: {"bytes_written":598,"delete_count":0,"lbm_write_time_us":164,"lbm_writes_lt_1ms":4,"reinsert_count":0,"spinlock_wait_cycles":23424,"update_count":11}
I20260929 02:01:04.921693 17096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 044313d0ac094292808355c152d310aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.922518 17091 tablet_service.cc:1511] Processing CreateTablet for tablet 7f3e8d6962a940868e1ab2caa707cff1 (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.922881 17091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f3e8d6962a940868e1ab2caa707cff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.923799 17098 tablet_service.cc:1511] Processing CreateTablet for tablet a3f4dfb792b946728e051ae1791cb75a (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.924130 17098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3f4dfb792b946728e051ae1791cb75a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:04.925565 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1666 bytes on disk
I20260929 02:01:04.925895 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1666 bytes on disk
I20260929 02:01:04.926120 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1666 bytes on disk
I20260929 02:01:04.926719 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20260929 02:01:04.927151 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.927541 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.932082 17398 consensus_queue.cc:1048] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:04.932145 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.108s user 0.006s sys 0.010s Metrics: {"dirs.queue_time_us":15931,"dirs.run_cpu_time_us":765,"dirs.run_wall_time_us":13167,"lbm_write_time_us":388,"lbm_writes_lt_1ms":7,"mutex_wait_us":23109,"num_input_rowsets":0,"thread_start_us":49639,"threads_started":2}
I20260929 02:01:04.938424 17218 tablet_service.cc:1511] Processing CreateTablet for tablet 20a08b5cc6ae46ebb4696c9f3c83c5fa (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:04.945138 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.089s user 0.004s sys 0.000s Metrics: {"bytes_written":497,"delete_count":0,"lbm_write_time_us":183,"lbm_writes_lt_1ms":4,"mutex_wait_us":83837,"reinsert_count":0,"update_count":5}
I20260929 02:01:04.946895 17463 tablet_bootstrap.cc:492] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:04.949280 17463 tablet_bootstrap.cc:654] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:04.955427 17463 tablet_bootstrap.cc:492] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:04.955915 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1666 bytes on disk
I20260929 02:01:04.956203 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1666 bytes on disk
I20260929 02:01:04.956435 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 1666 bytes on disk
I20260929 02:01:04.957024 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":86,"lbm_reads_lt_1ms":4}
I20260929 02:01:04.957436 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.957710 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:04.955598 17463 ts_tablet_manager.cc:1449] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.003s
I20260929 02:01:04.959745 17463 raft_consensus.cc:359] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.959985 17463 raft_consensus.cc:385] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:04.960084 17463 raft_consensus.cc:740] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:04.961750 17463 consensus_queue.cc:260] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:04.963717 17463 ts_tablet_manager.cc:1480] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260929 02:01:04.964808 17463 tablet_bootstrap.cc:492] T 85b63af0846742c99534e9bd449181d3 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:04.975867 17369 consensus_queue.cc:1048] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:04.982582 17463 tablet_bootstrap.cc:654] T 85b63af0846742c99534e9bd449181d3 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:04.994303 17348 consensus_queue.cc:1048] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:04.993320 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.156s user 0.047s sys 0.048s Metrics: {"bytes_written":683,"cfile_cache_hit":36,"cfile_cache_hit_bytes":11093,"cfile_cache_miss":54,"cfile_cache_miss_bytes":15477,"cfile_init":15,"delete_count":0,"delta_iterators_relevant":21,"dirs.queue_time_us":3788,"dirs.run_cpu_time_us":1929,"dirs.run_wall_time_us":72942,"drs_written":1,"lbm_read_time_us":2095,"lbm_reads_lt_1ms":114,"lbm_write_time_us":976,"lbm_writes_lt_1ms":31,"mutex_wait_us":217,"num_input_rowsets":3,"peak_mem_usage":98304,"reinsert_count":0,"rows_written":37,"spinlock_wait_cycles":1536,"thread_start_us":63733,"threads_started":3,"update_count":16}
W20260929 02:01:04.995021 17055 consensus_peers.cc:597] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc -> Peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Couldn't send request to peer d103dd4823044faebe46100a0dfbee88. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: b2d556cada174f83a633a7a8836826ce. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:01:05.001096 17463 tablet_bootstrap.cc:492] T 85b63af0846742c99534e9bd449181d3 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:05.001286 17463 ts_tablet_manager.cc:1449] T 85b63af0846742c99534e9bd449181d3 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.037s user 0.006s sys 0.006s
I20260929 02:01:05.002265 17463 raft_consensus.cc:359] T 85b63af0846742c99534e9bd449181d3 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.002538 17463 raft_consensus.cc:385] T 85b63af0846742c99534e9bd449181d3 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.002631 17463 raft_consensus.cc:740] T 85b63af0846742c99534e9bd449181d3 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.002924 17463 consensus_queue.cc:260] T 85b63af0846742c99534e9bd449181d3 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.003638 17463 ts_tablet_manager.cc:1480] T 85b63af0846742c99534e9bd449181d3 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 02:01:05.003988 17463 tablet_bootstrap.cc:492] T ea01c17d8d4b405ab885e7eec91b5cbc P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:05.006268 17463 tablet_bootstrap.cc:654] T ea01c17d8d4b405ab885e7eec91b5cbc P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.009313 17463 tablet_bootstrap.cc:492] T ea01c17d8d4b405ab885e7eec91b5cbc P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:05.009474 17463 ts_tablet_manager.cc:1449] T ea01c17d8d4b405ab885e7eec91b5cbc P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 02:01:05.010318 17463 raft_consensus.cc:359] T ea01c17d8d4b405ab885e7eec91b5cbc P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.010524 17463 raft_consensus.cc:385] T ea01c17d8d4b405ab885e7eec91b5cbc P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.010634 17463 raft_consensus.cc:740] T ea01c17d8d4b405ab885e7eec91b5cbc P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.012708 17463 consensus_queue.cc:260] T ea01c17d8d4b405ab885e7eec91b5cbc P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.013375 17463 ts_tablet_manager.cc:1480] T ea01c17d8d4b405ab885e7eec91b5cbc P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.004s user 0.000s sys 0.003s
I20260929 02:01:05.013794 17463 tablet_bootstrap.cc:492] T 2e9a65b8f4be4f1d99eef1517d20743c P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:05.016054 17369 raft_consensus.cc:493] T ea01c17d8d4b405ab885e7eec91b5cbc P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:05.016243 17369 raft_consensus.cc:515] T ea01c17d8d4b405ab885e7eec91b5cbc P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.016986 17369 leader_election.cc:290] T ea01c17d8d4b405ab885e7eec91b5cbc P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:05.017099 17463 tablet_bootstrap.cc:654] T 2e9a65b8f4be4f1d99eef1517d20743c P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
W20260929 02:01:05.018424 17055 leader_election.cc:343] T ea01c17d8d4b405ab885e7eec91b5cbc P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Not found: Tablet not found: ea01c17d8d4b405ab885e7eec91b5cbc
I20260929 02:01:05.041751 17458 ts_tablet_manager.cc:1975] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:05.041950 17458 log.cc:1199] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/7a962f60729242a1bcebe88c89f8f895
I20260929 02:01:05.042531 17458 ts_tablet_manager.cc:1996] T 7a962f60729242a1bcebe88c89f8f895 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:05.046665 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 7a962f60729242a1bcebe88c89f8f895 (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:05.052875 17458 tablet_replica.cc:333] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:05.053194 17458 raft_consensus.cc:2243] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:05.053664 17458 raft_consensus.cc:2272] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:05.054935 17458 ts_tablet_manager.cc:1962] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:05.057922 17369 raft_consensus.cc:493] T 85b63af0846742c99534e9bd449181d3 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:05.058122 17369 raft_consensus.cc:515] T 85b63af0846742c99534e9bd449181d3 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.059017 17369 leader_election.cc:290] T 85b63af0846742c99534e9bd449181d3 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
W20260929 02:01:05.060371 17055 leader_election.cc:343] T 85b63af0846742c99534e9bd449181d3 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Not found: Tablet not found: 85b63af0846742c99534e9bd449181d3
I20260929 02:01:05.061529 17456 ts_tablet_manager.cc:1975] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:05.061702 17456 log.cc:1199] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/c64e85b42e0d4618885975ac4125d4f2
I20260929 02:01:05.063227 17456 ts_tablet_manager.cc:1996] T c64e85b42e0d4618885975ac4125d4f2 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:05.068073 17458 ts_tablet_manager.cc:1975] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:05.068257 17458 log.cc:1199] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/b2d556cada174f83a633a7a8836826ce
I20260929 02:01:05.068727 17458 ts_tablet_manager.cc:1996] T b2d556cada174f83a633a7a8836826ce P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:05.068562 16955 tablet_service.cc:1511] Processing CreateTablet for tablet a3f4dfb792b946728e051ae1791cb75a (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:05.069180 16955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3f4dfb792b946728e051ae1791cb75a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:05.070067 17458 tablet_replica.cc:333] T b591026c764b48a9af227b7be6dd0ee6 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:05.070331 17458 raft_consensus.cc:2243] T b591026c764b48a9af227b7be6dd0ee6 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:05.070570 17458 raft_consensus.cc:2272] T b591026c764b48a9af227b7be6dd0ee6 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:05.076242 17456 tablet_replica.cc:333] T 0aa899b966dd481088d18119ee57d81a P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:05.076548 17456 raft_consensus.cc:2243] T 0aa899b966dd481088d18119ee57d81a P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:05.076773 17456 raft_consensus.cc:2272] T 0aa899b966dd481088d18119ee57d81a P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:05.077605 17456 ts_tablet_manager.cc:1962] T 0aa899b966dd481088d18119ee57d81a P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:05.078418 17458 ts_tablet_manager.cc:1962] T b591026c764b48a9af227b7be6dd0ee6 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 02:01:05.082860 16919 consensus_peers.cc:597] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f -> Peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Couldn't send request to peer 8668afa5b5ca4ebdabc812d0789de7bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2bc3dbd1c82e4d2cb9577c51d9b19224. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:01:05.084874 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea01c17d8d4b405ab885e7eec91b5cbc" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
W20260929 02:01:05.086035 17056 leader_election.cc:343] T 85b63af0846742c99534e9bd449181d3 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Not found: Tablet not found: 85b63af0846742c99534e9bd449181d3
I20260929 02:01:05.086248 17056 leader_election.cc:304] T 85b63af0846742c99534e9bd449181d3 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc; no voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88
W20260929 02:01:05.086843 17056 leader_election.cc:343] T ea01c17d8d4b405ab885e7eec91b5cbc P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:05.086987 17056 leader_election.cc:304] T ea01c17d8d4b405ab885e7eec91b5cbc P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc; no voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88
I20260929 02:01:05.087436 17369 consensus_queue.cc:1048] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:05.088225 17369 raft_consensus.cc:2749] T 85b63af0846742c99534e9bd449181d3 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:05.088500 17369 raft_consensus.cc:2749] T ea01c17d8d4b405ab885e7eec91b5cbc P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:05.090112 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet c64e85b42e0d4618885975ac4125d4f2 (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:05.091899 17458 ts_tablet_manager.cc:1975] T b591026c764b48a9af227b7be6dd0ee6 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:05.092065 17458 log.cc:1199] T b591026c764b48a9af227b7be6dd0ee6 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/b591026c764b48a9af227b7be6dd0ee6
I20260929 02:01:05.102355 17458 ts_tablet_manager.cc:1996] T b591026c764b48a9af227b7be6dd0ee6 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:05.105479 17458 tablet_replica.cc:333] T 8601b0ded51b49d0abeb95f7e5fedf8c P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:05.105823 17458 raft_consensus.cc:2243] T 8601b0ded51b49d0abeb95f7e5fedf8c P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:05.106055 17458 raft_consensus.cc:2272] T 8601b0ded51b49d0abeb95f7e5fedf8c P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:05.110077 17458 ts_tablet_manager.cc:1962] T 8601b0ded51b49d0abeb95f7e5fedf8c P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:05.117792 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet b2d556cada174f83a633a7a8836826ce (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:05.130904 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet b591026c764b48a9af227b7be6dd0ee6 (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:05.137122 17223 tablet_service.cc:1511] Processing CreateTablet for tablet 2e9a65b8f4be4f1d99eef1517d20743c (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:05.137797 17227 tablet_service.cc:1511] Processing CreateTablet for tablet 00f69d45340f4d578a7f3b810c6f9fec (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:05.138130 17458 ts_tablet_manager.cc:1975] T 8601b0ded51b49d0abeb95f7e5fedf8c P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:05.138231 17226 tablet_service.cc:1511] Processing CreateTablet for tablet a4083e6b1e594ee88efbd7e1f04f70e0 (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:05.138351 17458 log.cc:1199] T 8601b0ded51b49d0abeb95f7e5fedf8c P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/8601b0ded51b49d0abeb95f7e5fedf8c
I20260929 02:01:05.138654 17221 tablet_service.cc:1511] Processing CreateTablet for tablet d5da16034f2244a2b335f89f6ab7caa9 (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:05.138929 17458 ts_tablet_manager.cc:1996] T 8601b0ded51b49d0abeb95f7e5fedf8c P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:05.139050 17231 tablet_service.cc:1511] Processing CreateTablet for tablet 85b63af0846742c99534e9bd449181d3 (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:05.139483 17228 tablet_service.cc:1511] Processing CreateTablet for tablet 5e7d7ccf81494fec9e066824c69abd61 (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:05.139881 17224 tablet_service.cc:1511] Processing CreateTablet for tablet 044313d0ac094292808355c152d310aa (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:05.140280 17229 tablet_service.cc:1511] Processing CreateTablet for tablet 7f3e8d6962a940868e1ab2caa707cff1 (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:05.140373 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 8601b0ded51b49d0abeb95f7e5fedf8c (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:05.140703 17230 tablet_service.cc:1511] Processing CreateTablet for tablet a3f4dfb792b946728e051ae1791cb75a (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:05.141090 17222 tablet_service.cc:1511] Processing CreateTablet for tablet 3f0398c024bf4cc48ec1670fc847c7ab (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:05.141485 17219 tablet_service.cc:1511] Processing CreateTablet for tablet 2486911d1a704e159ed5ad88ebdd82c1 (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:05.141870 17220 tablet_service.cc:1511] Processing CreateTablet for tablet 7822796e1a01411aaa1c2b394cc27263 (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:05.142249 17225 tablet_service.cc:1511] Processing CreateTablet for tablet 8cd30e2805234d9fadc0d2ad839a6bc3 (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:05.142649 17232 tablet_service.cc:1511] Processing CreateTablet for tablet 9603392111a04ed395f159d69ac4801c (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:05.143268 17458 tablet_replica.cc:333] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:05.143347 17218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 20a08b5cc6ae46ebb4696c9f3c83c5fa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:05.148540 17233 tablet_service.cc:1511] Processing CreateTablet for tablet ea01c17d8d4b405ab885e7eec91b5cbc (DEFAULT_TABLE table=table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:05.149135 17233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ea01c17d8d4b405ab885e7eec91b5cbc. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:05.152532 17433 consensus_queue.cc:1048] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.176s
I20260929 02:01:05.153503 17232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 9603392111a04ed395f159d69ac4801c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:05.155249 17225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8cd30e2805234d9fadc0d2ad839a6bc3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:05.160760 17220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7822796e1a01411aaa1c2b394cc27263. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:05.164681 17219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2486911d1a704e159ed5ad88ebdd82c1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:05.168625 17230 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3f4dfb792b946728e051ae1791cb75a. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:05.172673 17231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 85b63af0846742c99534e9bd449181d3. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:05.175621 17458 raft_consensus.cc:2243] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:05.176011 17458 pending_rounds.cc:70] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc: Trying to abort 1 pending ops.
I20260929 02:01:05.176091 17458 pending_rounds.cc:77] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7334491195034345472 op_type: NO_OP noop_request { }
I20260929 02:01:05.176280 17458 raft_consensus.cc:2889] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260929 02:01:05.176636 17222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3f0398c024bf4cc48ec1670fc847c7ab. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:05.176410 17458 raft_consensus.cc:2272] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:05.179359 17458 ts_tablet_manager.cc:1962] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:05.180155 17228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e7d7ccf81494fec9e066824c69abd61. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:05.184708 17227 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 00f69d45340f4d578a7f3b810c6f9fec. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:05.188637 17221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d5da16034f2244a2b335f89f6ab7caa9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:05.192332 17229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7f3e8d6962a940868e1ab2caa707cff1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:05.196681 17224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 044313d0ac094292808355c152d310aa. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:05.200627 17226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a4083e6b1e594ee88efbd7e1f04f70e0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:05.205159 17223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e9a65b8f4be4f1d99eef1517d20743c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:05.206427 17458 ts_tablet_manager.cc:1975] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:05.206561 17458 log.cc:1199] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/3cb85eb3aa494f21bc8e506004318650
I20260929 02:01:05.206909 17458 ts_tablet_manager.cc:1996] T 3cb85eb3aa494f21bc8e506004318650 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:05.208088 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 3cb85eb3aa494f21bc8e506004318650 (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
W20260929 02:01:05.214783 17189 consensus_peers.cc:597] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88 -> Peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Couldn't send request to peer 97df55a0361841319adfae27c9787b2f. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:01:05.218924 17348 consensus_queue.cc:1048] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.139s
I20260929 02:01:05.225157 17458 tablet_replica.cc:333] T c7189d73aa7c43b99293f2da0e47338b P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:05.225457 17458 raft_consensus.cc:2243] T c7189d73aa7c43b99293f2da0e47338b P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:05.225730 17458 raft_consensus.cc:2272] T c7189d73aa7c43b99293f2da0e47338b P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:05.228003 17458 ts_tablet_manager.cc:1962] T c7189d73aa7c43b99293f2da0e47338b P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:05.259336 17458 ts_tablet_manager.cc:1975] T c7189d73aa7c43b99293f2da0e47338b P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:05.259544 17458 log.cc:1199] T c7189d73aa7c43b99293f2da0e47338b P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/c7189d73aa7c43b99293f2da0e47338b
I20260929 02:01:05.260012 17458 ts_tablet_manager.cc:1996] T c7189d73aa7c43b99293f2da0e47338b P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:05.270382 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet c7189d73aa7c43b99293f2da0e47338b (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:05.271459 17458 tablet_replica.cc:333] T c64e85b42e0d4618885975ac4125d4f2 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:05.271687 17458 raft_consensus.cc:2243] T c64e85b42e0d4618885975ac4125d4f2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:05.271883 17458 raft_consensus.cc:2272] T c64e85b42e0d4618885975ac4125d4f2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:05.273612 17458 ts_tablet_manager.cc:1962] T c64e85b42e0d4618885975ac4125d4f2 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:05.295951 17458 ts_tablet_manager.cc:1975] T c64e85b42e0d4618885975ac4125d4f2 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:05.296200 17458 log.cc:1199] T c64e85b42e0d4618885975ac4125d4f2 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/c64e85b42e0d4618885975ac4125d4f2
I20260929 02:01:05.311985 17456 ts_tablet_manager.cc:1975] T 0aa899b966dd481088d18119ee57d81a P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
W20260929 02:01:05.316147 17056 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.71.65:43625, user_credentials={real_user=slave}} blocked reactor thread for 229277us
I20260929 02:01:05.334010 17457 ts_tablet_manager.cc:1962] T b591026c764b48a9af227b7be6dd0ee6 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:05.352954 17458 ts_tablet_manager.cc:1996] T c64e85b42e0d4618885975ac4125d4f2 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:05.354652 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet c64e85b42e0d4618885975ac4125d4f2 (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:05.356067 17458 tablet_replica.cc:333] T 0aa899b966dd481088d18119ee57d81a P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:05.356396 17458 raft_consensus.cc:2243] T 0aa899b966dd481088d18119ee57d81a P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:05.356714 17458 raft_consensus.cc:2272] T 0aa899b966dd481088d18119ee57d81a P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:05.358016 17458 ts_tablet_manager.cc:1962] T 0aa899b966dd481088d18119ee57d81a P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 02:01:05.359125 17186 consensus_peers.cc:597] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88 -> Peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Couldn't send request to peer 8668afa5b5ca4ebdabc812d0789de7bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 8601b0ded51b49d0abeb95f7e5fedf8c. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:01:05.362206 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.544s user 0.116s sys 0.233s Metrics: {"bytes_written":683,"cfile_cache_hit":19,"cfile_cache_hit_bytes":9339,"cfile_cache_miss":49,"cfile_cache_miss_bytes":16659,"cfile_init":15,"delete_count":0,"delta_iterators_relevant":18,"dirs.queue_time_us":11457,"dirs.run_cpu_time_us":1613,"dirs.run_wall_time_us":10267,"drs_written":1,"lbm_read_time_us":6763,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":108,"lbm_write_time_us":1242,"lbm_writes_lt_1ms":31,"mutex_wait_us":23685,"num_input_rowsets":2,"peak_mem_usage":65536,"reinsert_count":0,"rows_written":37,"thread_start_us":410174,"threads_started":2,"update_count":16}
I20260929 02:01:05.370262 17458 ts_tablet_manager.cc:1975] T 0aa899b966dd481088d18119ee57d81a P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:05.370443 17458 log.cc:1199] T 0aa899b966dd481088d18119ee57d81a P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/0aa899b966dd481088d18119ee57d81a
I20260929 02:01:05.371127 17458 ts_tablet_manager.cc:1996] T 0aa899b966dd481088d18119ee57d81a P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:05.372651 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 0aa899b966dd481088d18119ee57d81a (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:05.392732 17458 tablet_replica.cc:333] T 43f983a2802b4ad3871d743b42b5deea P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:05.393071 17458 raft_consensus.cc:2243] T 43f983a2802b4ad3871d743b42b5deea P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:05.393327 17458 raft_consensus.cc:2272] T 43f983a2802b4ad3871d743b42b5deea P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:05.394764 17458 ts_tablet_manager.cc:1962] T 43f983a2802b4ad3871d743b42b5deea P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 02:01:05.323052 16921 consensus_peers.cc:597] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f -> Peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Couldn't send request to peer d103dd4823044faebe46100a0dfbee88. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2bc3dbd1c82e4d2cb9577c51d9b19224. This is attempt 1: this message will repeat every 5th retry.
W20260929 02:01:05.396184 16921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.71.67:44233, user_credentials={real_user=slave}} blocked reactor thread for 73145.4us
I20260929 02:01:05.396382 17456 log.cc:1199] T 0aa899b966dd481088d18119ee57d81a P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/0aa899b966dd481088d18119ee57d81a
I20260929 02:01:05.397634 17456 ts_tablet_manager.cc:1996] T 0aa899b966dd481088d18119ee57d81a P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:05.400419 17456 tablet_replica.cc:333] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:05.400714 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 0aa899b966dd481088d18119ee57d81a (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:05.402621 17458 ts_tablet_manager.cc:1975] T 43f983a2802b4ad3871d743b42b5deea P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:05.402773 17458 log.cc:1199] T 43f983a2802b4ad3871d743b42b5deea P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/43f983a2802b4ad3871d743b42b5deea
I20260929 02:01:05.403227 17458 ts_tablet_manager.cc:1996] T 43f983a2802b4ad3871d743b42b5deea P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:05.404299 17458 tablet_replica.cc:333] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:05.404424 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 43f983a2802b4ad3871d743b42b5deea (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:05.404526 17458 raft_consensus.cc:2243] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:05.404861 17458 raft_consensus.cc:2272] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:05.405644 17456 raft_consensus.cc:2243] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:05.405700 17458 ts_tablet_manager.cc:1962] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:05.407280 17456 raft_consensus.cc:2272] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:05.410431 17456 ts_tablet_manager.cc:1962] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:05.354635 17462 tablet_bootstrap.cc:492] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:05.415691 17462 tablet_bootstrap.cc:654] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.421624 17398 raft_consensus.cc:493] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:05.421864 17398 raft_consensus.cc:515] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.422523 17398 leader_election.cc:290] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:05.423246 17458 ts_tablet_manager.cc:1975] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:05.423372 17458 log.cc:1199] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/4d9e051c702b42bc9392c5dcae42265d
I20260929 02:01:05.423516 17252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a08b5cc6ae46ebb4696c9f3c83c5fa" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:05.423796 17458 ts_tablet_manager.cc:1996] T 4d9e051c702b42bc9392c5dcae42265d P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:05.424027 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a08b5cc6ae46ebb4696c9f3c83c5fa" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:05.424839 17481 tablet_bootstrap.cc:492] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:05.425033 17458 tablet_replica.cc:333] T 37e07f15e43c49a4bf5ab1334dd85d22 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:05.425174 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 4d9e051c702b42bc9392c5dcae42265d (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:05.425232 17458 raft_consensus.cc:2243] T 37e07f15e43c49a4bf5ab1334dd85d22 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:05.425395 17458 raft_consensus.cc:2272] T 37e07f15e43c49a4bf5ab1334dd85d22 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 02:01:05.425755 16800 catalog_manager.cc:5030] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): delete failed for tablet 2bc3dbd1c82e4d2cb9577c51d9b19224 because the tablet was not found. No further retry: Not found: Tablet not found: 2bc3dbd1c82e4d2cb9577c51d9b19224
I20260929 02:01:05.426679 17481 tablet_bootstrap.cc:654] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
W20260929 02:01:05.426884 17055 leader_election.cc:343] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
W20260929 02:01:05.429646 17056 leader_election.cc:343] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:05.429809 17056 leader_election.cc:304] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc; no voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88
I20260929 02:01:05.434819 17398 raft_consensus.cc:2749] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:05.435333 17458 ts_tablet_manager.cc:1962] T 37e07f15e43c49a4bf5ab1334dd85d22 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:05.441565 17463 tablet_bootstrap.cc:492] T 2e9a65b8f4be4f1d99eef1517d20743c P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:05.441877 17463 ts_tablet_manager.cc:1449] T 2e9a65b8f4be4f1d99eef1517d20743c P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.428s user 0.082s sys 0.184s
I20260929 02:01:05.442797 17463 raft_consensus.cc:359] T 2e9a65b8f4be4f1d99eef1517d20743c P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.442997 17462 tablet_bootstrap.cc:492] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:05.443149 17463 raft_consensus.cc:385] T 2e9a65b8f4be4f1d99eef1517d20743c P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.443240 17463 raft_consensus.cc:740] T 2e9a65b8f4be4f1d99eef1517d20743c P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.443276 17462 ts_tablet_manager.cc:1449] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.089s user 0.007s sys 0.000s
I20260929 02:01:05.443547 17463 consensus_queue.cc:260] T 2e9a65b8f4be4f1d99eef1517d20743c P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.444429 17462 raft_consensus.cc:359] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.444718 17462 raft_consensus.cc:385] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.444868 17462 raft_consensus.cc:740] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.445572 17463 ts_tablet_manager.cc:1480] T 2e9a65b8f4be4f1d99eef1517d20743c P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260929 02:01:05.445938 17463 tablet_bootstrap.cc:492] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:05.445191 17462 consensus_queue.cc:260] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.451314 17463 tablet_bootstrap.cc:654] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.453840 17462 ts_tablet_manager.cc:1480] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20260929 02:01:05.457660 17462 tablet_bootstrap.cc:492] T 5e7d7ccf81494fec9e066824c69abd61 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:05.460146 17456 ts_tablet_manager.cc:1975] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:05.460318 17456 log.cc:1199] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/43f983a2802b4ad3871d743b42b5deea
I20260929 02:01:05.460919 17462 tablet_bootstrap.cc:654] T 5e7d7ccf81494fec9e066824c69abd61 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.462401 17481 tablet_bootstrap.cc:492] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:05.462610 17481 ts_tablet_manager.cc:1449] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.038s user 0.013s sys 0.017s
I20260929 02:01:05.463765 17481 raft_consensus.cc:359] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.463896 17458 ts_tablet_manager.cc:1975] T 37e07f15e43c49a4bf5ab1334dd85d22 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:05.464048 17458 log.cc:1199] T 37e07f15e43c49a4bf5ab1334dd85d22 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/37e07f15e43c49a4bf5ab1334dd85d22
I20260929 02:01:05.464020 17481 raft_consensus.cc:385] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.464155 17481 raft_consensus.cc:740] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.464529 17458 ts_tablet_manager.cc:1996] T 37e07f15e43c49a4bf5ab1334dd85d22 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:05.464567 17481 consensus_queue.cc:260] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.466130 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 37e07f15e43c49a4bf5ab1334dd85d22 (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:05.446595 17457 ts_tablet_manager.cc:1975] T b591026c764b48a9af227b7be6dd0ee6 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:05.467728 17457 log.cc:1199] T b591026c764b48a9af227b7be6dd0ee6 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/b591026c764b48a9af227b7be6dd0ee6
I20260929 02:01:05.467720 17456 ts_tablet_manager.cc:1996] T 43f983a2802b4ad3871d743b42b5deea P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:05.468204 17457 ts_tablet_manager.cc:1996] T b591026c764b48a9af227b7be6dd0ee6 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:05.469257 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet b591026c764b48a9af227b7be6dd0ee6 (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:05.469600 17457 tablet_replica.cc:333] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:05.469813 17457 raft_consensus.cc:2243] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:05.470209 17457 raft_consensus.cc:2272] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:05.470429 17463 tablet_bootstrap.cc:492] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:05.470585 17463 ts_tablet_manager.cc:1449] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.025s user 0.001s sys 0.003s
I20260929 02:01:05.470829 17457 ts_tablet_manager.cc:1962] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:05.471475 17463 raft_consensus.cc:359] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.471684 17463 raft_consensus.cc:385] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.471771 17463 raft_consensus.cc:740] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.472009 17463 consensus_queue.cc:260] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.472613 17463 ts_tablet_manager.cc:1480] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:01:05.472957 17463 tablet_bootstrap.cc:492] T d5da16034f2244a2b335f89f6ab7caa9 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:05.474835 17463 tablet_bootstrap.cc:654] T d5da16034f2244a2b335f89f6ab7caa9 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.476810 17481 ts_tablet_manager.cc:1480] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.014s user 0.003s sys 0.000s
I20260929 02:01:05.479058 17481 tablet_bootstrap.cc:492] T 00f69d45340f4d578a7f3b810c6f9fec P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:05.481619 17481 tablet_bootstrap.cc:654] T 00f69d45340f4d578a7f3b810c6f9fec P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.483480 17463 tablet_bootstrap.cc:492] T d5da16034f2244a2b335f89f6ab7caa9 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:05.483659 17463 ts_tablet_manager.cc:1449] T d5da16034f2244a2b335f89f6ab7caa9 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.001s
I20260929 02:01:05.484216 17462 tablet_bootstrap.cc:492] T 5e7d7ccf81494fec9e066824c69abd61 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:05.484486 17462 ts_tablet_manager.cc:1449] T 5e7d7ccf81494fec9e066824c69abd61 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.027s user 0.004s sys 0.000s
I20260929 02:01:05.484467 17463 raft_consensus.cc:359] T d5da16034f2244a2b335f89f6ab7caa9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.484683 17463 raft_consensus.cc:385] T d5da16034f2244a2b335f89f6ab7caa9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.484777 17463 raft_consensus.cc:740] T d5da16034f2244a2b335f89f6ab7caa9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.485059 17463 consensus_queue.cc:260] T d5da16034f2244a2b335f89f6ab7caa9 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.487483 17463 ts_tablet_manager.cc:1480] T d5da16034f2244a2b335f89f6ab7caa9 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260929 02:01:05.489537 17463 tablet_bootstrap.cc:492] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:05.490051 17462 raft_consensus.cc:359] T 5e7d7ccf81494fec9e066824c69abd61 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.490378 17462 raft_consensus.cc:385] T 5e7d7ccf81494fec9e066824c69abd61 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.490514 17462 raft_consensus.cc:740] T 5e7d7ccf81494fec9e066824c69abd61 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.490887 17462 consensus_queue.cc:260] T 5e7d7ccf81494fec9e066824c69abd61 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.491374 17463 tablet_bootstrap.cc:654] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.492990 17462 ts_tablet_manager.cc:1480] T 5e7d7ccf81494fec9e066824c69abd61 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260929 02:01:05.493886 17462 tablet_bootstrap.cc:492] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:05.494084 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 43f983a2802b4ad3871d743b42b5deea (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:05.496479 17457 ts_tablet_manager.cc:1975] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:05.496656 17457 log.cc:1199] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/8601b0ded51b49d0abeb95f7e5fedf8c
I20260929 02:01:05.497231 17457 ts_tablet_manager.cc:1996] T 8601b0ded51b49d0abeb95f7e5fedf8c P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:05.493547 17456 tablet_replica.cc:333] T 4d9e051c702b42bc9392c5dcae42265d P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:05.498163 17456 raft_consensus.cc:2243] T 4d9e051c702b42bc9392c5dcae42265d P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:05.498440 17456 raft_consensus.cc:2272] T 4d9e051c702b42bc9392c5dcae42265d P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:05.500289 17456 ts_tablet_manager.cc:1962] T 4d9e051c702b42bc9392c5dcae42265d P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:05.502043 17481 tablet_bootstrap.cc:492] T 00f69d45340f4d578a7f3b810c6f9fec P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:05.502215 17481 ts_tablet_manager.cc:1449] T 00f69d45340f4d578a7f3b810c6f9fec P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.023s user 0.007s sys 0.011s
I20260929 02:01:05.503043 17481 raft_consensus.cc:359] T 00f69d45340f4d578a7f3b810c6f9fec P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.503258 17481 raft_consensus.cc:385] T 00f69d45340f4d578a7f3b810c6f9fec P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.503350 17481 raft_consensus.cc:740] T 00f69d45340f4d578a7f3b810c6f9fec P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.503669 17481 consensus_queue.cc:260] T 00f69d45340f4d578a7f3b810c6f9fec P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.501497 17462 tablet_bootstrap.cc:654] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.506851 17481 ts_tablet_manager.cc:1480] T 00f69d45340f4d578a7f3b810c6f9fec P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260929 02:01:05.507357 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 8601b0ded51b49d0abeb95f7e5fedf8c (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:05.509402 17481 tablet_bootstrap.cc:492] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:05.510653 17457 tablet_replica.cc:333] T 3cb85eb3aa494f21bc8e506004318650 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:05.510673 17463 tablet_bootstrap.cc:492] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:05.510911 17457 raft_consensus.cc:2243] T 3cb85eb3aa494f21bc8e506004318650 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:05.510968 17463 ts_tablet_manager.cc:1449] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.008s
I20260929 02:01:05.511070 17457 pending_rounds.cc:70] T 3cb85eb3aa494f21bc8e506004318650 P d103dd4823044faebe46100a0dfbee88: Trying to abort 1 pending ops.
I20260929 02:01:05.511250 17457 pending_rounds.cc:77] T 3cb85eb3aa494f21bc8e506004318650 P d103dd4823044faebe46100a0dfbee88: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7334491195034345472 op_type: NO_OP noop_request { }
I20260929 02:01:05.511399 17457 raft_consensus.cc:2889] T 3cb85eb3aa494f21bc8e506004318650 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260929 02:01:05.511632 17481 tablet_bootstrap.cc:654] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.512392 17463 raft_consensus.cc:359] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.512655 17463 raft_consensus.cc:385] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.512830 17463 raft_consensus.cc:740] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.513239 17463 consensus_queue.cc:260] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.513928 17463 ts_tablet_manager.cc:1480] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:01:05.514246 17463 tablet_bootstrap.cc:492] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:05.516039 17463 tablet_bootstrap.cc:654] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.516958 17457 raft_consensus.cc:2272] T 3cb85eb3aa494f21bc8e506004318650 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:05.517869 17457 ts_tablet_manager.cc:1962] T 3cb85eb3aa494f21bc8e506004318650 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:05.528306 17456 ts_tablet_manager.cc:1975] T 4d9e051c702b42bc9392c5dcae42265d P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:05.532081 17456 log.cc:1199] T 4d9e051c702b42bc9392c5dcae42265d P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/4d9e051c702b42bc9392c5dcae42265d
I20260929 02:01:05.532923 17456 ts_tablet_manager.cc:1996] T 4d9e051c702b42bc9392c5dcae42265d P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:05.534303 17462 tablet_bootstrap.cc:492] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:05.534401 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 4d9e051c702b42bc9392c5dcae42265d (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:05.534585 17462 ts_tablet_manager.cc:1449] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.041s user 0.009s sys 0.000s
I20260929 02:01:05.535396 17463 tablet_bootstrap.cc:492] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:05.535593 17463 ts_tablet_manager.cc:1449] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.003s
I20260929 02:01:05.535519 17462 raft_consensus.cc:359] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.535838 17462 raft_consensus.cc:385] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.535969 17462 raft_consensus.cc:740] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.536327 17462 consensus_queue.cc:260] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.536469 17463 raft_consensus.cc:359] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.536667 17463 raft_consensus.cc:385] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.536743 17463 raft_consensus.cc:740] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.536971 17463 consensus_queue.cc:260] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.537480 17463 ts_tablet_manager.cc:1480] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:01:05.539930 17457 ts_tablet_manager.cc:1975] T 3cb85eb3aa494f21bc8e506004318650 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:05.540071 17457 log.cc:1199] T 3cb85eb3aa494f21bc8e506004318650 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/3cb85eb3aa494f21bc8e506004318650
I20260929 02:01:05.541566 17481 tablet_bootstrap.cc:492] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:05.541726 17481 ts_tablet_manager.cc:1449] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.032s user 0.009s sys 0.001s
I20260929 02:01:05.542266 17462 ts_tablet_manager.cc:1480] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.008s user 0.002s sys 0.000s
I20260929 02:01:05.542446 17481 raft_consensus.cc:359] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.542620 17481 raft_consensus.cc:385] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.542697 17481 raft_consensus.cc:740] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.537786 17463 tablet_bootstrap.cc:492] T 044313d0ac094292808355c152d310aa P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:05.543010 17481 consensus_queue.cc:260] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.544806 17463 tablet_bootstrap.cc:654] T 044313d0ac094292808355c152d310aa P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.545742 17462 tablet_bootstrap.cc:492] T 044313d0ac094292808355c152d310aa P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:05.546236 17456 tablet_replica.cc:333] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:05.546480 17456 raft_consensus.cc:2243] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:05.546846 17481 ts_tablet_manager.cc:1480] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260929 02:01:05.547605 17456 raft_consensus.cc:2272] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:05.548874 17456 ts_tablet_manager.cc:1962] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:05.549348 17481 tablet_bootstrap.cc:492] T 7822796e1a01411aaa1c2b394cc27263 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:05.550510 17462 tablet_bootstrap.cc:654] T 044313d0ac094292808355c152d310aa P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.551137 17457 ts_tablet_manager.cc:1996] T 3cb85eb3aa494f21bc8e506004318650 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:05.552229 17457 tablet_replica.cc:333] T c7189d73aa7c43b99293f2da0e47338b P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:05.552373 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 3cb85eb3aa494f21bc8e506004318650 (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:05.552680 17457 raft_consensus.cc:2243] T c7189d73aa7c43b99293f2da0e47338b P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:05.552875 17457 raft_consensus.cc:2272] T c7189d73aa7c43b99293f2da0e47338b P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:05.553476 17457 ts_tablet_manager.cc:1962] T c7189d73aa7c43b99293f2da0e47338b P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:05.555776 17481 tablet_bootstrap.cc:654] T 7822796e1a01411aaa1c2b394cc27263 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.560034 17462 tablet_bootstrap.cc:492] T 044313d0ac094292808355c152d310aa P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:05.560276 17462 ts_tablet_manager.cc:1449] T 044313d0ac094292808355c152d310aa P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.015s user 0.010s sys 0.000s
I20260929 02:01:05.561164 17462 raft_consensus.cc:359] T 044313d0ac094292808355c152d310aa P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.561456 17462 raft_consensus.cc:385] T 044313d0ac094292808355c152d310aa P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.561589 17462 raft_consensus.cc:740] T 044313d0ac094292808355c152d310aa P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.561931 17462 consensus_queue.cc:260] T 044313d0ac094292808355c152d310aa P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.562678 17456 ts_tablet_manager.cc:1975] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:05.562837 17456 log.cc:1199] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/2bc3dbd1c82e4d2cb9577c51d9b19224
I20260929 02:01:05.563336 17456 ts_tablet_manager.cc:1996] T 2bc3dbd1c82e4d2cb9577c51d9b19224 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:05.564391 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 2bc3dbd1c82e4d2cb9577c51d9b19224 (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:05.564921 17462 ts_tablet_manager.cc:1480] T 044313d0ac094292808355c152d310aa P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.005s user 0.000s sys 0.002s
I20260929 02:01:05.565562 17456 tablet_replica.cc:333] T 37e07f15e43c49a4bf5ab1334dd85d22 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:05.565757 17456 raft_consensus.cc:2243] T 37e07f15e43c49a4bf5ab1334dd85d22 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:05.566164 17462 tablet_bootstrap.cc:492] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:05.566475 17457 ts_tablet_manager.cc:1975] T c7189d73aa7c43b99293f2da0e47338b P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:05.566642 17457 log.cc:1199] T c7189d73aa7c43b99293f2da0e47338b P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/c7189d73aa7c43b99293f2da0e47338b
I20260929 02:01:05.567062 17457 ts_tablet_manager.cc:1996] T c7189d73aa7c43b99293f2da0e47338b P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:05.567047 17456 raft_consensus.cc:2272] T 37e07f15e43c49a4bf5ab1334dd85d22 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:05.568234 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet c7189d73aa7c43b99293f2da0e47338b (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:05.568404 17410 raft_consensus.cc:493] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:05.568558 17410 raft_consensus.cc:515] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.569113 17463 tablet_bootstrap.cc:492] T 044313d0ac094292808355c152d310aa P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:05.569283 17463 ts_tablet_manager.cc:1449] T 044313d0ac094292808355c152d310aa P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.004s
I20260929 02:01:05.569298 17410 leader_election.cc:290] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:05.570086 17120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a08b5cc6ae46ebb4696c9f3c83c5fa" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:05.570171 17463 raft_consensus.cc:359] T 044313d0ac094292808355c152d310aa P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.570263 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a08b5cc6ae46ebb4696c9f3c83c5fa" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:05.570307 17120 raft_consensus.cc:2468] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:05.570377 17463 raft_consensus.cc:385] T 044313d0ac094292808355c152d310aa P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.570469 17463 raft_consensus.cc:740] T 044313d0ac094292808355c152d310aa P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.570748 17463 consensus_queue.cc:260] T 044313d0ac094292808355c152d310aa P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
W20260929 02:01:05.571062 16921 leader_election.cc:343] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:05.571509 16919 leader_election.cc:304] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters: d103dd4823044faebe46100a0dfbee88
I20260929 02:01:05.572333 17410 raft_consensus.cc:2804] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:05.572472 17410 raft_consensus.cc:493] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:05.572572 17410 raft_consensus.cc:3060] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.572646 17456 ts_tablet_manager.cc:1962] T 37e07f15e43c49a4bf5ab1334dd85d22 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:05.573530 17463 ts_tablet_manager.cc:1480] T 044313d0ac094292808355c152d310aa P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 02:01:05.575248 17463 tablet_bootstrap.cc:492] T a3f4dfb792b946728e051ae1791cb75a P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:05.576253 17457 tablet_replica.cc:333] T c64e85b42e0d4618885975ac4125d4f2 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:05.576480 17457 raft_consensus.cc:2243] T c64e85b42e0d4618885975ac4125d4f2 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:05.576781 17457 raft_consensus.cc:2272] T c64e85b42e0d4618885975ac4125d4f2 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:05.577418 17410 raft_consensus.cc:515] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.577615 17457 ts_tablet_manager.cc:1962] T c64e85b42e0d4618885975ac4125d4f2 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:05.577675 17463 tablet_bootstrap.cc:654] T a3f4dfb792b946728e051ae1791cb75a P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.580670 17120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a08b5cc6ae46ebb4696c9f3c83c5fa" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:05.580905 17120 raft_consensus.cc:3060] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.581295 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a08b5cc6ae46ebb4696c9f3c83c5fa" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
W20260929 02:01:05.582134 16921 leader_election.cc:343] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:05.582338 17410 leader_election.cc:290] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Requested vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:05.582832 17462 tablet_bootstrap.cc:654] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.583987 17120 raft_consensus.cc:2468] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:05.584563 16919 leader_election.cc:304] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters: d103dd4823044faebe46100a0dfbee88
I20260929 02:01:05.585165 17410 raft_consensus.cc:2804] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:05.586513 17410 raft_consensus.cc:697] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Becoming Leader. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Running, Role: LEADER
I20260929 02:01:05.586951 17410 consensus_queue.cc:237] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.587352 17481 tablet_bootstrap.cc:492] T 7822796e1a01411aaa1c2b394cc27263 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:05.587524 17481 ts_tablet_manager.cc:1449] T 7822796e1a01411aaa1c2b394cc27263 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.038s user 0.001s sys 0.013s
I20260929 02:01:05.587822 17463 tablet_bootstrap.cc:492] T a3f4dfb792b946728e051ae1791cb75a P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:05.587987 17463 ts_tablet_manager.cc:1449] T a3f4dfb792b946728e051ae1791cb75a P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.013s user 0.010s sys 0.001s
I20260929 02:01:05.588277 17481 raft_consensus.cc:359] T 7822796e1a01411aaa1c2b394cc27263 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.588510 17481 raft_consensus.cc:385] T 7822796e1a01411aaa1c2b394cc27263 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.588591 17481 raft_consensus.cc:740] T 7822796e1a01411aaa1c2b394cc27263 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.588851 17481 consensus_queue.cc:260] T 7822796e1a01411aaa1c2b394cc27263 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.588878 17463 raft_consensus.cc:359] T a3f4dfb792b946728e051ae1791cb75a P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.589102 17463 raft_consensus.cc:385] T a3f4dfb792b946728e051ae1791cb75a P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.589188 17463 raft_consensus.cc:740] T a3f4dfb792b946728e051ae1791cb75a P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.589532 17463 consensus_queue.cc:260] T a3f4dfb792b946728e051ae1791cb75a P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.592036 17481 ts_tablet_manager.cc:1480] T 7822796e1a01411aaa1c2b394cc27263 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 02:01:05.592517 17463 ts_tablet_manager.cc:1480] T a3f4dfb792b946728e051ae1791cb75a P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.004s user 0.003s sys 0.000s
I20260929 02:01:05.594007 17481 tablet_bootstrap.cc:492] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:05.596016 17481 tablet_bootstrap.cc:654] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.596055 17463 tablet_bootstrap.cc:492] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:05.597393 17456 ts_tablet_manager.cc:1975] T 37e07f15e43c49a4bf5ab1334dd85d22 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:05.597546 17456 log.cc:1199] T 37e07f15e43c49a4bf5ab1334dd85d22 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/37e07f15e43c49a4bf5ab1334dd85d22
I20260929 02:01:05.597998 17456 ts_tablet_manager.cc:1996] T 37e07f15e43c49a4bf5ab1334dd85d22 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:05.598292 17463 tablet_bootstrap.cc:654] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.599190 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 37e07f15e43c49a4bf5ab1334dd85d22 (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:05.600479 17462 tablet_bootstrap.cc:492] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:05.600944 17463 tablet_bootstrap.cc:492] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:05.601100 17463 ts_tablet_manager.cc:1449] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.005s user 0.003s sys 0.000s
I20260929 02:01:05.601307 17462 ts_tablet_manager.cc:1449] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.035s user 0.000s sys 0.007s
I20260929 02:01:05.601883 17457 ts_tablet_manager.cc:1975] T c64e85b42e0d4618885975ac4125d4f2 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:05.602020 17457 log.cc:1199] T c64e85b42e0d4618885975ac4125d4f2 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/c64e85b42e0d4618885975ac4125d4f2
I20260929 02:01:05.602003 17463 raft_consensus.cc:359] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.602226 17463 raft_consensus.cc:385] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.602372 17463 raft_consensus.cc:740] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.602526 17457 ts_tablet_manager.cc:1996] T c64e85b42e0d4618885975ac4125d4f2 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:05.602720 17463 consensus_queue.cc:260] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.603960 17463 ts_tablet_manager.cc:1480] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:01:05.604285 17463 tablet_bootstrap.cc:492] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:05.605290 17462 raft_consensus.cc:359] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
W20260929 02:01:05.605634 17186 consensus_peers.cc:597] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88 -> Peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Couldn't send request to peer 8668afa5b5ca4ebdabc812d0789de7bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 37e07f15e43c49a4bf5ab1334dd85d22. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:01:05.608186 17462 raft_consensus.cc:385] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.608479 17462 raft_consensus.cc:740] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.608896 17462 consensus_queue.cc:260] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.609262 16819 catalog_manager.cc:5838] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97df55a0361841319adfae27c9787b2f (127.0.71.65). New cstate: current_term: 1 leader_uuid: "97df55a0361841319adfae27c9787b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:05.610244 17462 ts_tablet_manager.cc:1480] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.009s user 0.000s sys 0.002s
I20260929 02:01:05.610575 17462 tablet_bootstrap.cc:492] T 9603392111a04ed395f159d69ac4801c P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:05.612401 17463 tablet_bootstrap.cc:654] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.613540 17462 tablet_bootstrap.cc:654] T 9603392111a04ed395f159d69ac4801c P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.619242 17462 tablet_bootstrap.cc:492] T 9603392111a04ed395f159d69ac4801c P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:05.619519 17462 ts_tablet_manager.cc:1449] T 9603392111a04ed395f159d69ac4801c P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.009s user 0.000s sys 0.006s
I20260929 02:01:05.620553 17462 raft_consensus.cc:359] T 9603392111a04ed395f159d69ac4801c P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.620903 17462 raft_consensus.cc:385] T 9603392111a04ed395f159d69ac4801c P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.621050 17462 raft_consensus.cc:740] T 9603392111a04ed395f159d69ac4801c P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.621500 17462 consensus_queue.cc:260] T 9603392111a04ed395f159d69ac4801c P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.622053 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet c64e85b42e0d4618885975ac4125d4f2 (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:05.622166 17457 tablet_replica.cc:333] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:05.622473 17457 raft_consensus.cc:2243] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:05.622939 17462 ts_tablet_manager.cc:1480] T 9603392111a04ed395f159d69ac4801c P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:01:05.622953 17457 raft_consensus.cc:2272] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:05.623240 17462 tablet_bootstrap.cc:492] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:05.624193 17457 ts_tablet_manager.cc:1962] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:05.624707 17481 tablet_bootstrap.cc:492] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:05.624905 17481 ts_tablet_manager.cc:1449] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.031s user 0.005s sys 0.001s
I20260929 02:01:05.625778 17481 raft_consensus.cc:359] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.626039 17481 raft_consensus.cc:385] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.626181 17481 raft_consensus.cc:740] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.626497 17481 consensus_queue.cc:260] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.627893 17481 ts_tablet_manager.cc:1480] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 02:01:05.628245 17481 tablet_bootstrap.cc:492] T 2486911d1a704e159ed5ad88ebdd82c1 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:05.630224 17481 tablet_bootstrap.cc:654] T 2486911d1a704e159ed5ad88ebdd82c1 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.632098 17457 ts_tablet_manager.cc:1975] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:05.632236 17457 log.cc:1199] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/0aa899b966dd481088d18119ee57d81a
I20260929 02:01:05.632695 17457 ts_tablet_manager.cc:1996] T 0aa899b966dd481088d18119ee57d81a P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:05.633900 17457 tablet_replica.cc:333] T 43f983a2802b4ad3871d743b42b5deea P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:05.634141 17457 raft_consensus.cc:2243] T 43f983a2802b4ad3871d743b42b5deea P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:05.634371 17457 raft_consensus.cc:2272] T 43f983a2802b4ad3871d743b42b5deea P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:05.634572 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 0aa899b966dd481088d18119ee57d81a (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:05.634976 17457 ts_tablet_manager.cc:1962] T 43f983a2802b4ad3871d743b42b5deea P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:05.636277 17462 tablet_bootstrap.cc:654] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.639470 17463 tablet_bootstrap.cc:492] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:05.639691 17463 ts_tablet_manager.cc:1449] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.035s user 0.020s sys 0.000s
I20260929 02:01:05.640655 17463 raft_consensus.cc:359] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.640884 17463 raft_consensus.cc:385] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.640962 17463 raft_consensus.cc:740] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.641628 17463 consensus_queue.cc:260] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.643323 17457 ts_tablet_manager.cc:1975] T 43f983a2802b4ad3871d743b42b5deea P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:05.643460 17457 log.cc:1199] T 43f983a2802b4ad3871d743b42b5deea P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/43f983a2802b4ad3871d743b42b5deea
I20260929 02:01:05.643942 17457 ts_tablet_manager.cc:1996] T 43f983a2802b4ad3871d743b42b5deea P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:05.644919 17457 tablet_replica.cc:333] T 4d9e051c702b42bc9392c5dcae42265d P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:05.645057 17481 tablet_bootstrap.cc:492] T 2486911d1a704e159ed5ad88ebdd82c1 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:05.645035 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 43f983a2802b4ad3871d743b42b5deea (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:05.645147 17457 raft_consensus.cc:2243] T 4d9e051c702b42bc9392c5dcae42265d P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:05.645267 17481 ts_tablet_manager.cc:1449] T 2486911d1a704e159ed5ad88ebdd82c1 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.017s user 0.009s sys 0.001s
I20260929 02:01:05.645296 17457 pending_rounds.cc:70] T 4d9e051c702b42bc9392c5dcae42265d P d103dd4823044faebe46100a0dfbee88: Trying to abort 1 pending ops.
I20260929 02:01:05.645737 17457 pending_rounds.cc:77] T 4d9e051c702b42bc9392c5dcae42265d P d103dd4823044faebe46100a0dfbee88: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7334491195051352064 op_type: NO_OP noop_request { }
I20260929 02:01:05.645887 17457 raft_consensus.cc:2889] T 4d9e051c702b42bc9392c5dcae42265d P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260929 02:01:05.646021 17457 raft_consensus.cc:2272] T 4d9e051c702b42bc9392c5dcae42265d P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:05.646071 17481 raft_consensus.cc:359] T 2486911d1a704e159ed5ad88ebdd82c1 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.646261 17481 raft_consensus.cc:385] T 2486911d1a704e159ed5ad88ebdd82c1 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.646347 17481 raft_consensus.cc:740] T 2486911d1a704e159ed5ad88ebdd82c1 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.646540 17463 ts_tablet_manager.cc:1480] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260929 02:01:05.646607 17481 consensus_queue.cc:260] T 2486911d1a704e159ed5ad88ebdd82c1 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.646935 17463 tablet_bootstrap.cc:492] T 7822796e1a01411aaa1c2b394cc27263 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:05.647279 17457 ts_tablet_manager.cc:1962] T 4d9e051c702b42bc9392c5dcae42265d P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:05.648901 17463 tablet_bootstrap.cc:654] T 7822796e1a01411aaa1c2b394cc27263 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.650888 17481 ts_tablet_manager.cc:1480] T 2486911d1a704e159ed5ad88ebdd82c1 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260929 02:01:05.651677 17462 tablet_bootstrap.cc:492] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:05.651868 17481 tablet_bootstrap.cc:492] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:05.651975 17462 ts_tablet_manager.cc:1449] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.029s user 0.004s sys 0.000s
I20260929 02:01:05.652889 17462 raft_consensus.cc:359] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.653141 17462 raft_consensus.cc:385] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.653870 17398 raft_consensus.cc:493] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:05.654039 17398 raft_consensus.cc:515] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.654150 17462 raft_consensus.cc:740] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.654330 17481 tablet_bootstrap.cc:654] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.654579 17462 consensus_queue.cc:260] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.654763 17398 leader_election.cc:290] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:05.655587 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cd30e2805234d9fadc0d2ad839a6bc3" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:05.656270 17462 ts_tablet_manager.cc:1480] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
W20260929 02:01:05.656551 17056 leader_election.cc:343] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:05.657245 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cd30e2805234d9fadc0d2ad839a6bc3" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
W20260929 02:01:05.658053 17055 leader_election.cc:343] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:05.658195 17055 leader_election.cc:304] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc; no voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88
I20260929 02:01:05.658660 17398 raft_consensus.cc:2749] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:05.658918 17457 ts_tablet_manager.cc:1975] T 4d9e051c702b42bc9392c5dcae42265d P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:05.659037 17457 log.cc:1199] T 4d9e051c702b42bc9392c5dcae42265d P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/4d9e051c702b42bc9392c5dcae42265d
I20260929 02:01:05.659484 17457 ts_tablet_manager.cc:1996] T 4d9e051c702b42bc9392c5dcae42265d P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:05.660950 17462 tablet_bootstrap.cc:492] T 3f0398c024bf4cc48ec1670fc847c7ab P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:05.662950 17462 tablet_bootstrap.cc:654] T 3f0398c024bf4cc48ec1670fc847c7ab P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.665066 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 4d9e051c702b42bc9392c5dcae42265d (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:05.665159 17457 tablet_replica.cc:333] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:05.665361 17457 raft_consensus.cc:2243] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Raft consensus shutting down.
W20260929 02:01:05.665658 16802 catalog_manager.cc:5030] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): delete failed for tablet 2bc3dbd1c82e4d2cb9577c51d9b19224 because the tablet was not found. No further retry: Not found: Tablet not found: 2bc3dbd1c82e4d2cb9577c51d9b19224
I20260929 02:01:05.665947 17382 raft_consensus.cc:493] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:05.666108 17382 raft_consensus.cc:515] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.666975 17382 leader_election.cc:290] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:05.667742 17398 raft_consensus.cc:493] T d5da16034f2244a2b335f89f6ab7caa9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:05.667707 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e7d7ccf81494fec9e066824c69abd61" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:05.667905 16986 raft_consensus.cc:2468] T 5e7d7ccf81494fec9e066824c69abd61 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 0.
I20260929 02:01:05.667934 17398 raft_consensus.cc:515] T d5da16034f2244a2b335f89f6ab7caa9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.668362 17189 leader_election.cc:304] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:05.668656 17398 leader_election.cc:290] T d5da16034f2244a2b335f89f6ab7caa9 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:05.669106 17120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e7d7ccf81494fec9e066824c69abd61" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
W20260929 02:01:05.669603 17186 leader_election.cc:343] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:05.669602 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5da16034f2244a2b335f89f6ab7caa9" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
W20260929 02:01:05.670192 17056 leader_election.cc:343] T d5da16034f2244a2b335f89f6ab7caa9 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:05.670192 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5da16034f2244a2b335f89f6ab7caa9" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:05.670888 17382 raft_consensus.cc:2804] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:05.671028 17382 raft_consensus.cc:493] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:05.671139 17382 raft_consensus.cc:3060] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.673645 17410 raft_consensus.cc:493] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:05.673833 17410 raft_consensus.cc:515] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.674523 17410 leader_election.cc:290] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:05.675220 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e9a65b8f4be4f1d99eef1517d20743c" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
W20260929 02:01:05.675750 16921 leader_election.cc:343] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
W20260929 02:01:05.676236 17055 leader_election.cc:343] T d5da16034f2244a2b335f89f6ab7caa9 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:05.676394 17120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e9a65b8f4be4f1d99eef1517d20743c" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:05.676467 17055 leader_election.cc:304] T d5da16034f2244a2b335f89f6ab7caa9 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc; no voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88
I20260929 02:01:05.676618 17120 raft_consensus.cc:2468] T 2e9a65b8f4be4f1d99eef1517d20743c P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:05.677130 16919 leader_election.cc:304] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters: d103dd4823044faebe46100a0dfbee88
I20260929 02:01:05.677436 17398 raft_consensus.cc:2749] T d5da16034f2244a2b335f89f6ab7caa9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:05.677747 17410 raft_consensus.cc:2804] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:05.677788 17382 raft_consensus.cc:515] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.677872 17410 raft_consensus.cc:493] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:05.677995 17410 raft_consensus.cc:3060] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.678925 17463 tablet_bootstrap.cc:492] T 7822796e1a01411aaa1c2b394cc27263 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:05.679016 17120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e7d7ccf81494fec9e066824c69abd61" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:05.679126 17463 ts_tablet_manager.cc:1449] T 7822796e1a01411aaa1c2b394cc27263 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.032s user 0.005s sys 0.002s
W20260929 02:01:05.679577 17186 leader_election.cc:343] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:05.679803 17382 leader_election.cc:290] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:05.679968 17457 raft_consensus.cc:2272] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:05.680109 17463 raft_consensus.cc:359] T 7822796e1a01411aaa1c2b394cc27263 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.680155 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e7d7ccf81494fec9e066824c69abd61" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:05.680297 17463 raft_consensus.cc:385] T 7822796e1a01411aaa1c2b394cc27263 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.680402 16986 raft_consensus.cc:3060] T 5e7d7ccf81494fec9e066824c69abd61 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.681111 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.681414 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.681699 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.681989 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.682607 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.682868 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.680392 17463 raft_consensus.cc:740] T 7822796e1a01411aaa1c2b394cc27263 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.683065 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.683341 17463 consensus_queue.cc:260] T 7822796e1a01411aaa1c2b394cc27263 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.683458 16986 raft_consensus.cc:2468] T 5e7d7ccf81494fec9e066824c69abd61 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 1.
I20260929 02:01:05.683954 17189 leader_election.cc:304] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc
I20260929 02:01:05.684736 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.155200
I20260929 02:01:05.684948 17463 ts_tablet_manager.cc:1480] T 7822796e1a01411aaa1c2b394cc27263 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260929 02:01:05.685030 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.155200
I20260929 02:01:05.685240 17463 tablet_bootstrap.cc:492] T 9603392111a04ed395f159d69ac4801c P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:05.685315 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.155200
I20260929 02:01:05.685712 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.685936 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.686381 17410 raft_consensus.cc:515] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.687004 17463 tablet_bootstrap.cc:654] T 9603392111a04ed395f159d69ac4801c P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.687409 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.687512 17120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e9a65b8f4be4f1d99eef1517d20743c" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:05.687630 17410 leader_election.cc:290] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Requested vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:05.687762 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.687795 17120 raft_consensus.cc:3060] T 2e9a65b8f4be4f1d99eef1517d20743c P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.688124 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.689744 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.690171 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.155200
I20260929 02:01:05.690413 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.155200
I20260929 02:01:05.690657 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.691058 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.691419 17120 raft_consensus.cc:2468] T 2e9a65b8f4be4f1d99eef1517d20743c P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:05.693094 16919 leader_election.cc:304] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:05.695230 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e9a65b8f4be4f1d99eef1517d20743c" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
W20260929 02:01:05.696089 16921 leader_election.cc:343] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:05.699865 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.018s user 0.000s sys 0.004s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":183,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:05.700534 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.013s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":165,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:05.700897 17410 raft_consensus.cc:2804] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:05.700946 17457 ts_tablet_manager.cc:1962] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:05.701113 17382 raft_consensus.cc:2804] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:05.701301 17382 raft_consensus.cc:697] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Becoming Leader. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Running, Role: LEADER
I20260929 02:01:05.701665 17382 consensus_queue.cc:237] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.702610 17410 raft_consensus.cc:697] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Becoming Leader. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Running, Role: LEADER
I20260929 02:01:05.702787 17462 tablet_bootstrap.cc:492] T 3f0398c024bf4cc48ec1670fc847c7ab P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:05.702901 17410 consensus_queue.cc:237] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.703135 17462 ts_tablet_manager.cc:1449] T 3f0398c024bf4cc48ec1670fc847c7ab P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.042s user 0.004s sys 0.000s
I20260929 02:01:05.704238 17462 raft_consensus.cc:359] T 3f0398c024bf4cc48ec1670fc847c7ab P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.704754 17462 raft_consensus.cc:385] T 3f0398c024bf4cc48ec1670fc847c7ab P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.704918 17462 raft_consensus.cc:740] T 3f0398c024bf4cc48ec1670fc847c7ab P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.705305 17462 consensus_queue.cc:260] T 3f0398c024bf4cc48ec1670fc847c7ab P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.709262 17462 ts_tablet_manager.cc:1480] T 3f0398c024bf4cc48ec1670fc847c7ab P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260929 02:01:05.709731 17462 tablet_bootstrap.cc:492] T 00f69d45340f4d578a7f3b810c6f9fec P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:05.712378 17462 tablet_bootstrap.cc:654] T 00f69d45340f4d578a7f3b810c6f9fec P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.713235 17481 tablet_bootstrap.cc:492] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:05.713402 17481 ts_tablet_manager.cc:1449] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.062s user 0.014s sys 0.010s
I20260929 02:01:05.714295 17481 raft_consensus.cc:359] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.714504 17481 raft_consensus.cc:385] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.714597 17481 raft_consensus.cc:740] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.714610 17463 tablet_bootstrap.cc:492] T 9603392111a04ed395f159d69ac4801c P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:05.714788 17463 ts_tablet_manager.cc:1449] T 9603392111a04ed395f159d69ac4801c P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.030s user 0.021s sys 0.006s
I20260929 02:01:05.715606 17463 raft_consensus.cc:359] T 9603392111a04ed395f159d69ac4801c P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.715826 17463 raft_consensus.cc:385] T 9603392111a04ed395f159d69ac4801c P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.715921 17463 raft_consensus.cc:740] T 9603392111a04ed395f159d69ac4801c P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.716193 17463 consensus_queue.cc:260] T 9603392111a04ed395f159d69ac4801c P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.717482 17463 ts_tablet_manager.cc:1480] T 9603392111a04ed395f159d69ac4801c P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260929 02:01:05.717783 17463 tablet_bootstrap.cc:492] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:05.719038 17481 consensus_queue.cc:260] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.720062 17463 tablet_bootstrap.cc:654] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.721491 17481 ts_tablet_manager.cc:1480] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.008s user 0.000s sys 0.002s
I20260929 02:01:05.721850 17481 tablet_bootstrap.cc:492] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:05.723846 17481 tablet_bootstrap.cc:654] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.719817 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.029s user 0.011s sys 0.000s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":8676,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":32163,"lbm_read_time_us":370,"lbm_reads_lt_1ms":23,"lbm_write_time_us":321,"lbm_writes_lt_1ms":9,"mutex_wait_us":2914,"peak_mem_usage":32768,"reinsert_count":0,"update_count":17}
I20260929 02:01:05.725802 17457 ts_tablet_manager.cc:1975] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:05.725986 17457 log.cc:1199] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/37e07f15e43c49a4bf5ab1334dd85d22
I20260929 02:01:05.726526 17457 ts_tablet_manager.cc:1996] T 37e07f15e43c49a4bf5ab1334dd85d22 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:05.728048 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 37e07f15e43c49a4bf5ab1334dd85d22 (table table-0 [id=b262d8c39bfe4f0abba8b155fe000610]) successfully deleted
I20260929 02:01:05.730988 17480 raft_consensus.cc:493] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:05.731146 17480 raft_consensus.cc:515] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.731997 17480 leader_election.cc:290] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:05.731254 16819 catalog_manager.cc:5838] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97df55a0361841319adfae27c9787b2f (127.0.71.65). New cstate: current_term: 1 leader_uuid: "97df55a0361841319adfae27c9787b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:05.732897 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3f4dfb792b946728e051ae1791cb75a" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:05.733080 17120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3f4dfb792b946728e051ae1791cb75a" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:05.733351 17120 raft_consensus.cc:2468] T a3f4dfb792b946728e051ae1791cb75a P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 0.
W20260929 02:01:05.733610 17189 leader_election.cc:343] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:05.736428 16818 catalog_manager.cc:5838] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88 reported cstate change: term changed from 0 to 1, leader changed from <none> to d103dd4823044faebe46100a0dfbee88 (127.0.71.67). New cstate: current_term: 1 leader_uuid: "d103dd4823044faebe46100a0dfbee88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: HEALTHY } } }
I20260929 02:01:05.737985 17186 leader_election.cc:304] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters: 97df55a0361841319adfae27c9787b2f
I20260929 02:01:05.740738 17382 raft_consensus.cc:2804] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:05.740876 17382 raft_consensus.cc:493] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:05.741014 17382 raft_consensus.cc:3060] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.744735 17382 raft_consensus.cc:515] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.745816 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3f4dfb792b946728e051ae1791cb75a" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
W20260929 02:01:05.746428 17189 leader_election.cc:343] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:05.747341 17120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3f4dfb792b946728e051ae1791cb75a" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:05.747592 17120 raft_consensus.cc:3060] T a3f4dfb792b946728e051ae1791cb75a P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.751760 17120 raft_consensus.cc:2468] T a3f4dfb792b946728e051ae1791cb75a P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 1.
I20260929 02:01:05.752347 17186 leader_election.cc:304] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters: 97df55a0361841319adfae27c9787b2f
I20260929 02:01:05.752677 17480 raft_consensus.cc:2804] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:05.752733 17382 leader_election.cc:290] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:05.752859 17480 raft_consensus.cc:697] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Becoming Leader. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Running, Role: LEADER
I20260929 02:01:05.753166 17480 consensus_queue.cc:237] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.754531 17462 tablet_bootstrap.cc:492] T 00f69d45340f4d578a7f3b810c6f9fec P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:05.754705 17462 ts_tablet_manager.cc:1449] T 00f69d45340f4d578a7f3b810c6f9fec P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.045s user 0.003s sys 0.004s
I20260929 02:01:05.755782 17462 raft_consensus.cc:359] T 00f69d45340f4d578a7f3b810c6f9fec P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.756007 17462 raft_consensus.cc:385] T 00f69d45340f4d578a7f3b810c6f9fec P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.756094 17462 raft_consensus.cc:740] T 00f69d45340f4d578a7f3b810c6f9fec P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.756318 17462 consensus_queue.cc:260] T 00f69d45340f4d578a7f3b810c6f9fec P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.756624 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.756969 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.756999 17462 ts_tablet_manager.cc:1480] T 00f69d45340f4d578a7f3b810c6f9fec P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:01:05.757296 17462 tablet_bootstrap.cc:492] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:05.758814 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.759549 17462 tablet_bootstrap.cc:654] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.763862 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.764225 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.764477 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.008s user 0.006s sys 0.000s Metrics: {"bytes_written":466,"delete_count":0,"lbm_write_time_us":176,"lbm_writes_lt_1ms":4,"reinsert_count":0,"spinlock_wait_cycles":3840,"update_count":2}
I20260929 02:01:05.765774 17463 tablet_bootstrap.cc:492] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:05.765944 17463 ts_tablet_manager.cc:1449] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.048s user 0.009s sys 0.018s
I20260929 02:01:05.766093 17462 tablet_bootstrap.cc:492] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:05.766252 17462 ts_tablet_manager.cc:1449] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.009s user 0.002s sys 0.004s
I20260929 02:01:05.766789 17463 raft_consensus.cc:359] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.766952 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.766981 17463 raft_consensus.cc:385] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.766991 17462 raft_consensus.cc:359] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.767079 17463 raft_consensus.cc:740] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.767194 17462 raft_consensus.cc:385] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.767277 17462 raft_consensus.cc:740] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.767402 17463 consensus_queue.cc:260] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.767539 17462 consensus_queue.cc:260] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.768601 17463 ts_tablet_manager.cc:1480] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.003s user 0.001s sys 0.002s
I20260929 02:01:05.768877 17463 tablet_bootstrap.cc:492] T 5e7d7ccf81494fec9e066824c69abd61 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:05.769004 17410 raft_consensus.cc:493] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:05.769137 17410 raft_consensus.cc:515] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.769824 17410 leader_election.cc:290] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:05.770648 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7822796e1a01411aaa1c2b394cc27263" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:05.770890 17253 raft_consensus.cc:2468] T 7822796e1a01411aaa1c2b394cc27263 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:05.771232 17463 tablet_bootstrap.cc:654] T 5e7d7ccf81494fec9e066824c69abd61 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.771356 16921 leader_election.cc:304] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:05.772058 17120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7822796e1a01411aaa1c2b394cc27263" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:05.772300 17120 raft_consensus.cc:2468] T 7822796e1a01411aaa1c2b394cc27263 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:05.776067 17410 raft_consensus.cc:2804] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:05.776223 17410 raft_consensus.cc:493] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:05.776329 17410 raft_consensus.cc:3060] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.780251 17410 raft_consensus.cc:515] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.781828 16818 catalog_manager.cc:5838] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88 reported cstate change: term changed from 0 to 1, leader changed from <none> to d103dd4823044faebe46100a0dfbee88 (127.0.71.67). New cstate: current_term: 1 leader_uuid: "d103dd4823044faebe46100a0dfbee88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: HEALTHY } } }
I20260929 02:01:05.786538 17120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7822796e1a01411aaa1c2b394cc27263" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:05.786815 17120 raft_consensus.cc:3060] T 7822796e1a01411aaa1c2b394cc27263 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.791005 17120 raft_consensus.cc:2468] T 7822796e1a01411aaa1c2b394cc27263 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:05.791580 16919 leader_election.cc:304] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:05.792073 17337 raft_consensus.cc:2804] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:05.792294 17337 raft_consensus.cc:697] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Becoming Leader. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Running, Role: LEADER
I20260929 02:01:05.792287 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.028s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":24993}
I20260929 02:01:05.792588 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.028s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":25303}
I20260929 02:01:05.792693 17337 consensus_queue.cc:237] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.793368 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.026s user 0.006s sys 0.001s Metrics: {"bytes_written":466,"delete_count":0,"lbm_write_time_us":212,"lbm_writes_lt_1ms":4,"mutex_wait_us":17466,"reinsert_count":0,"update_count":2}
I20260929 02:01:05.795866 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7822796e1a01411aaa1c2b394cc27263" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:05.796088 17253 raft_consensus.cc:3060] T 7822796e1a01411aaa1c2b394cc27263 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.798283 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.113s user 0.014s sys 0.000s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":8676,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":39539,"dirs.run_cpu_time_us":333,"dirs.run_wall_time_us":1762,"lbm_read_time_us":440,"lbm_reads_lt_1ms":23,"lbm_write_time_us":326,"lbm_writes_lt_1ms":9,"mutex_wait_us":14209,"peak_mem_usage":32768,"reinsert_count":0,"thread_start_us":51127,"threads_started":1,"update_count":17}
I20260929 02:01:05.799113 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.799453 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.799798 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.799896 17253 raft_consensus.cc:2468] T 7822796e1a01411aaa1c2b394cc27263 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:05.800107 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.800392 17410 leader_election.cc:290] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Requested vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:05.801585 17480 raft_consensus.cc:493] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:05.801724 17480 raft_consensus.cc:515] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.801997 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.802222 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.802412 17480 leader_election.cc:290] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:05.802450 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.802809 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:05.803079 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:05.803143 17462 ts_tablet_manager.cc:1480] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.037s user 0.001s sys 0.001s
I20260929 02:01:05.803345 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:05.803649 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.803772 17120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9603392111a04ed395f159d69ac4801c" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:05.803999 17120 raft_consensus.cc:2468] T 9603392111a04ed395f159d69ac4801c P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 0.
I20260929 02:01:05.804224 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.804243 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9603392111a04ed395f159d69ac4801c" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:05.804499 17186 leader_election.cc:304] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:05.804505 16986 raft_consensus.cc:2468] T 9603392111a04ed395f159d69ac4801c P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 0.
I20260929 02:01:05.804776 17480 raft_consensus.cc:2804] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:05.804894 17480 raft_consensus.cc:493] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:05.805008 17480 raft_consensus.cc:3060] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.805280 17462 tablet_bootstrap.cc:492] T a4083e6b1e594ee88efbd7e1f04f70e0 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:05.807777 17398 raft_consensus.cc:493] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:05.808029 17480 raft_consensus.cc:515] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.807991 17398 raft_consensus.cc:515] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.808781 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.808849 17398 leader_election.cc:290] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:05.809113 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.809417 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.809679 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.809811 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f0398c024bf4cc48ec1670fc847c7ab" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:05.809811 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f0398c024bf4cc48ec1670fc847c7ab" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:05.809989 16986 raft_consensus.cc:2468] T 3f0398c024bf4cc48ec1670fc847c7ab P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
W20260929 02:01:05.810328 17055 leader_election.cc:343] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:05.810464 17056 leader_election.cc:304] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters: d103dd4823044faebe46100a0dfbee88
I20260929 02:01:05.811048 17398 raft_consensus.cc:2804] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:05.811159 17398 raft_consensus.cc:493] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:05.811295 17398 raft_consensus.cc:3060] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.811533 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.811796 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.812018 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.812500 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:05.812599 17462 tablet_bootstrap.cc:654] T a4083e6b1e594ee88efbd7e1f04f70e0 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.812775 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:05.812368 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.012s user 0.002s sys 0.003s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":183,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:05.813036 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:05.813428 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.813652 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.814311 17480 leader_election.cc:290] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:05.814594 17398 raft_consensus.cc:515] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.814975 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9603392111a04ed395f159d69ac4801c" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:05.815157 16986 raft_consensus.cc:3060] T 9603392111a04ed395f159d69ac4801c P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.818327 16986 raft_consensus.cc:2468] T 9603392111a04ed395f159d69ac4801c P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 1.
I20260929 02:01:05.818603 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.009s user 0.001s sys 0.003s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":153,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:05.818825 17189 leader_election.cc:304] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:05.819294 17120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "9603392111a04ed395f159d69ac4801c" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:05.819491 17120 raft_consensus.cc:3060] T 9603392111a04ed395f159d69ac4801c P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.819422 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.016s user 0.007s sys 0.002s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":700,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":51,"lbm_read_time_us":179,"lbm_reads_lt_1ms":8,"lbm_write_time_us":195,"lbm_writes_lt_1ms":4,"mutex_wait_us":5282,"reinsert_count":0,"spinlock_wait_cycles":3128704,"update_count":3}
I20260929 02:01:05.820242 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f0398c024bf4cc48ec1670fc847c7ab" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:05.820628 16986 raft_consensus.cc:3060] T 3f0398c024bf4cc48ec1670fc847c7ab P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.822381 17480 raft_consensus.cc:2804] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:05.822451 17120 raft_consensus.cc:2468] T 9603392111a04ed395f159d69ac4801c P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 1.
I20260929 02:01:05.822535 17480 raft_consensus.cc:697] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Becoming Leader. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Running, Role: LEADER
I20260929 02:01:05.822834 17480 consensus_queue.cc:237] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.823189 17398 leader_election.cc:290] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:05.823628 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3f0398c024bf4cc48ec1670fc847c7ab" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
W20260929 02:01:05.824208 17055 leader_election.cc:343] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:05.824357 16986 raft_consensus.cc:2468] T 3f0398c024bf4cc48ec1670fc847c7ab P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:05.824842 17056 leader_election.cc:304] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters: d103dd4823044faebe46100a0dfbee88
I20260929 02:01:05.825228 17398 raft_consensus.cc:2804] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:05.825748 17398 raft_consensus.cc:697] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Becoming Leader. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Running, Role: LEADER
I20260929 02:01:05.826020 17398 consensus_queue.cc:237] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.826344 16818 catalog_manager.cc:5838] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97df55a0361841319adfae27c9787b2f (127.0.71.65). New cstate: current_term: 1 leader_uuid: "97df55a0361841319adfae27c9787b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:05.830665 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.017s user 0.005s sys 0.003s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":700,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":88,"lbm_read_time_us":153,"lbm_reads_lt_1ms":8,"lbm_write_time_us":168,"lbm_writes_lt_1ms":4,"mutex_wait_us":5266,"reinsert_count":0,"update_count":3}
I20260929 02:01:05.833875 17481 tablet_bootstrap.cc:492] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:05.834064 17481 ts_tablet_manager.cc:1449] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.112s user 0.020s sys 0.029s
I20260929 02:01:05.834863 17481 raft_consensus.cc:359] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.835121 17481 raft_consensus.cc:385] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.835194 17481 raft_consensus.cc:740] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.837702 17481 consensus_queue.cc:260] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.837870 17337 raft_consensus.cc:493] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:05.838338 17337 raft_consensus.cc:515] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.839825 17120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea01c17d8d4b405ab885e7eec91b5cbc" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:05.839998 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea01c17d8d4b405ab885e7eec91b5cbc" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:05.840081 17120 raft_consensus.cc:2468] T ea01c17d8d4b405ab885e7eec91b5cbc P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:05.840576 16919 leader_election.cc:304] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:05.841141 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
W20260929 02:01:05.841116 16921 leader_election.cc:343] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:05.841477 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.841569 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.841766 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.841895 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.842054 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.842162 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.842456 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.842238 16818 catalog_manager.cc:5838] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88 reported cstate change: term changed from 0 to 1, leader changed from <none> to d103dd4823044faebe46100a0dfbee88 (127.0.71.67). New cstate: current_term: 1 leader_uuid: "d103dd4823044faebe46100a0dfbee88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: HEALTHY } } }
I20260929 02:01:05.843956 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.844189 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.844249 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.844480 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.844602 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.844740 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.845057 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:05.845320 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:05.845592 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:05.845916 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.846134 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.846385 17410 raft_consensus.cc:2804] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:05.846478 17410 raft_consensus.cc:493] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:05.846572 17410 raft_consensus.cc:3060] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.848716 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:05.849017 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:05.849249 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:05.849607 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.849807 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.850106 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.850358 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.854267 17337 leader_election.cc:290] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:05.855813 17410 raft_consensus.cc:515] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.856261 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.014s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":158,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:05.856957 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.857245 17119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea01c17d8d4b405ab885e7eec91b5cbc" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:05.857478 17119 raft_consensus.cc:3060] T ea01c17d8d4b405ab885e7eec91b5cbc P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.857487 17463 tablet_bootstrap.cc:492] T 5e7d7ccf81494fec9e066824c69abd61 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:05.857786 17463 ts_tablet_manager.cc:1449] T 5e7d7ccf81494fec9e066824c69abd61 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.089s user 0.019s sys 0.027s
I20260929 02:01:05.858937 17463 raft_consensus.cc:359] T 5e7d7ccf81494fec9e066824c69abd61 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.859220 17463 raft_consensus.cc:385] T 5e7d7ccf81494fec9e066824c69abd61 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.859302 17463 raft_consensus.cc:740] T 5e7d7ccf81494fec9e066824c69abd61 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.859745 17463 consensus_queue.cc:260] T 5e7d7ccf81494fec9e066824c69abd61 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.860530 17463 ts_tablet_manager.cc:1480] T 5e7d7ccf81494fec9e066824c69abd61 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260929 02:01:05.861606 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ea01c17d8d4b405ab885e7eec91b5cbc" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
W20260929 02:01:05.862294 16921 leader_election.cc:343] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:05.862550 17410 leader_election.cc:290] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Requested vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:05.862923 17119 raft_consensus.cc:2468] T ea01c17d8d4b405ab885e7eec91b5cbc P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:05.863300 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.013s user 0.004s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":156,"lbm_writes_lt_1ms":4,"mutex_wait_us":8167,"reinsert_count":0,"update_count":1}
I20260929 02:01:05.863485 16919 leader_election.cc:304] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters: d103dd4823044faebe46100a0dfbee88
I20260929 02:01:05.863996 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.863945 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.022s user 0.003s sys 0.002s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":163,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:05.864408 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.008s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":7614}
I20260929 02:01:05.864645 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.864946 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.864794 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.015s user 0.000s sys 0.007s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":721,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":48,"dirs.run_cpu_time_us":133,"dirs.run_wall_time_us":579,"lbm_read_time_us":166,"lbm_reads_lt_1ms":8,"lbm_write_time_us":153,"lbm_writes_lt_1ms":4,"mutex_wait_us":1504,"reinsert_count":0,"update_count":4}
I20260929 02:01:05.865204 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.865324 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:05.865617 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:05.865878 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:05.866021 17410 raft_consensus.cc:2804] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:05.866151 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:05.866410 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.021s user 0.007s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":721,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"lbm_read_time_us":99,"lbm_reads_lt_1ms":4,"lbm_write_time_us":174,"lbm_writes_lt_1ms":4,"mutex_wait_us":6003,"reinsert_count":0,"update_count":4}
I20260929 02:01:05.868374 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:05.868623 17410 raft_consensus.cc:697] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Becoming Leader. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Running, Role: LEADER
I20260929 02:01:05.868991 17410 consensus_queue.cc:237] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.855918 17462 tablet_bootstrap.cc:492] T a4083e6b1e594ee88efbd7e1f04f70e0 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:05.870328 17462 ts_tablet_manager.cc:1449] T a4083e6b1e594ee88efbd7e1f04f70e0 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.067s user 0.000s sys 0.004s
I20260929 02:01:05.871277 17462 raft_consensus.cc:359] T a4083e6b1e594ee88efbd7e1f04f70e0 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.871456 17462 raft_consensus.cc:385] T a4083e6b1e594ee88efbd7e1f04f70e0 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.871544 17462 raft_consensus.cc:740] T a4083e6b1e594ee88efbd7e1f04f70e0 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.871797 17462 consensus_queue.cc:260] T a4083e6b1e594ee88efbd7e1f04f70e0 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.872181 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":6380}
I20260929 02:01:05.872112 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.002s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":186,"lbm_writes_lt_1ms":4,"mutex_wait_us":1097,"reinsert_count":0,"update_count":1}
I20260929 02:01:05.872546 17462 ts_tablet_manager.cc:1480] T a4083e6b1e594ee88efbd7e1f04f70e0 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 02:01:05.872632 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.872804 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":6822}
I20260929 02:01:05.872900 17462 tablet_bootstrap.cc:492] T 2486911d1a704e159ed5ad88ebdd82c1 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:05.873075 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.873473 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.875226 17462 tablet_bootstrap.cc:654] T 2486911d1a704e159ed5ad88ebdd82c1 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.877081 17481 ts_tablet_manager.cc:1480] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.043s user 0.001s sys 0.001s
I20260929 02:01:05.880092 17481 tablet_bootstrap.cc:492] T 3f0398c024bf4cc48ec1670fc847c7ab P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:05.880697 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.014s user 0.007s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":747,"cfile_init":2,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":2947,"lbm_read_time_us":217,"lbm_reads_lt_1ms":12,"lbm_write_time_us":168,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":5}
I20260929 02:01:05.882663 17481 tablet_bootstrap.cc:654] T 3f0398c024bf4cc48ec1670fc847c7ab P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.886555 17480 raft_consensus.cc:493] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:05.886816 17480 raft_consensus.cc:515] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.886664 16818 catalog_manager.cc:5838] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66). New cstate: current_term: 1 leader_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:05.886822 16819 catalog_manager.cc:5838] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97df55a0361841319adfae27c9787b2f (127.0.71.65). New cstate: current_term: 1 leader_uuid: "97df55a0361841319adfae27c9787b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:05.887487 17480 leader_election.cc:290] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:05.888260 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.018s user 0.006s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":747,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":47,"lbm_read_time_us":151,"lbm_reads_lt_1ms":8,"lbm_write_time_us":3633,"lbm_writes_1-10_ms":1,"lbm_writes_lt_1ms":3,"mutex_wait_us":2558,"reinsert_count":0,"update_count":5}
I20260929 02:01:05.888805 17120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044313d0ac094292808355c152d310aa" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:05.889046 17120 raft_consensus.cc:2468] T 044313d0ac094292808355c152d310aa P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 0.
I20260929 02:01:05.889313 17398 raft_consensus.cc:493] T 044313d0ac094292808355c152d310aa P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:05.889451 17398 raft_consensus.cc:515] T 044313d0ac094292808355c152d310aa P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.890204 17480 raft_consensus.cc:493] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:05.890282 17398 leader_election.cc:290] T 044313d0ac094292808355c152d310aa P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:05.890357 17480 raft_consensus.cc:515] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.890586 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044313d0ac094292808355c152d310aa" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:05.890770 17253 raft_consensus.cc:2468] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:05.891017 17480 leader_election.cc:290] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:05.891701 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044313d0ac094292808355c152d310aa" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:05.891765 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044313d0ac094292808355c152d310aa" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:05.891834 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a08b5cc6ae46ebb4696c9f3c83c5fa" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:05.891938 16986 raft_consensus.cc:2468] T 044313d0ac094292808355c152d310aa P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 0.
I20260929 02:01:05.891014 17186 leader_election.cc:304] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:05.892467 17480 raft_consensus.cc:2804] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:05.892592 17480 raft_consensus.cc:493] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:05.892685 17480 raft_consensus.cc:3060] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.892637 17055 leader_election.cc:304] T 044313d0ac094292808355c152d310aa P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters: 97df55a0361841319adfae27c9787b2f
I20260929 02:01:05.893059 17462 tablet_bootstrap.cc:492] T 2486911d1a704e159ed5ad88ebdd82c1 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:05.893221 17462 ts_tablet_manager.cc:1449] T 2486911d1a704e159ed5ad88ebdd82c1 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.020s user 0.000s sys 0.009s
I20260929 02:01:05.894421 17462 raft_consensus.cc:359] T 2486911d1a704e159ed5ad88ebdd82c1 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.894663 17462 raft_consensus.cc:385] T 2486911d1a704e159ed5ad88ebdd82c1 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.894742 17462 raft_consensus.cc:740] T 2486911d1a704e159ed5ad88ebdd82c1 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.895009 17462 consensus_queue.cc:260] T 2486911d1a704e159ed5ad88ebdd82c1 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.895594 17462 ts_tablet_manager.cc:1480] T 2486911d1a704e159ed5ad88ebdd82c1 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 02:01:05.895905 17462 tablet_bootstrap.cc:492] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:05.895825 17480 raft_consensus.cc:515] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.895905 17120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "20a08b5cc6ae46ebb4696c9f3c83c5fa" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:05.896140 17120 raft_consensus.cc:2393] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d103dd4823044faebe46100a0dfbee88 in current term 1: Already voted for candidate 97df55a0361841319adfae27c9787b2f in this term.
I20260929 02:01:05.896579 17186 leader_election.cc:304] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f
I20260929 02:01:05.896718 17480 leader_election.cc:290] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:05.896906 17382 raft_consensus.cc:3060] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.897411 17119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044313d0ac094292808355c152d310aa" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:05.897579 17119 raft_consensus.cc:3060] T 044313d0ac094292808355c152d310aa P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.897647 17462 tablet_bootstrap.cc:654] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.899086 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "044313d0ac094292808355c152d310aa" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:05.899320 16986 raft_consensus.cc:3060] T 044313d0ac094292808355c152d310aa P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.899669 17382 raft_consensus.cc:2749] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:05.900517 17119 raft_consensus.cc:2468] T 044313d0ac094292808355c152d310aa P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 1.
I20260929 02:01:05.900669 17475 raft_consensus.cc:2764] T 044313d0ac094292808355c152d310aa P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader pre-election decision vote started in defunct term 0: won
I20260929 02:01:05.901456 17186 leader_election.cc:304] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:05.901721 17480 raft_consensus.cc:2804] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:05.901841 17480 raft_consensus.cc:697] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Becoming Leader. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Running, Role: LEADER
I20260929 02:01:05.901935 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.902149 17480 consensus_queue.cc:237] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.902315 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.902613 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.902901 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.904691 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.905000 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.905243 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.906080 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.906397 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.906618 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.908553 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.908898 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.911365 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.100446
I20260929 02:01:05.912047 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.100446
I20260929 02:01:05.913072 17462 tablet_bootstrap.cc:492] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:05.913344 17462 ts_tablet_manager.cc:1449] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20260929 02:01:05.914184 17462 raft_consensus.cc:359] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.914374 17462 raft_consensus.cc:385] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.914463 17462 raft_consensus.cc:740] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.914678 17462 consensus_queue.cc:260] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.915902 17462 ts_tablet_manager.cc:1480] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:01:05.916262 17462 tablet_bootstrap.cc:492] T 7f3e8d6962a940868e1ab2caa707cff1 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:05.916558 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.100446
I20260929 02:01:05.916859 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:05.917091 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.917330 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.918130 17462 tablet_bootstrap.cc:654] T 7f3e8d6962a940868e1ab2caa707cff1 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.912424 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.100446
I20260929 02:01:05.920226 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:05.920270 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.920528 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.921948 16986 raft_consensus.cc:2468] T 044313d0ac094292808355c152d310aa P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 1.
I20260929 02:01:05.922850 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.016s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":138,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:05.923163 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.020s user 0.006s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":184,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:05.925634 17481 tablet_bootstrap.cc:492] T 3f0398c024bf4cc48ec1670fc847c7ab P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:05.925843 17481 ts_tablet_manager.cc:1449] T 3f0398c024bf4cc48ec1670fc847c7ab P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.046s user 0.014s sys 0.013s
I20260929 02:01:05.926937 17481 raft_consensus.cc:359] T 3f0398c024bf4cc48ec1670fc847c7ab P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.927202 17481 raft_consensus.cc:385] T 3f0398c024bf4cc48ec1670fc847c7ab P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.927340 17481 raft_consensus.cc:740] T 3f0398c024bf4cc48ec1670fc847c7ab P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.927608 17481 consensus_queue.cc:260] T 3f0398c024bf4cc48ec1670fc847c7ab P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.928184 17481 ts_tablet_manager.cc:1480] T 3f0398c024bf4cc48ec1670fc847c7ab P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 02:01:05.928586 17481 tablet_bootstrap.cc:492] T 2e9a65b8f4be4f1d99eef1517d20743c P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:05.929857 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.930153 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.930429 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.931538 17481 tablet_bootstrap.cc:654] T 2e9a65b8f4be4f1d99eef1517d20743c P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.932081 16817 catalog_manager.cc:5838] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88 reported cstate change: term changed from 0 to 1, leader changed from <none> to d103dd4823044faebe46100a0dfbee88 (127.0.71.67). New cstate: current_term: 1 leader_uuid: "d103dd4823044faebe46100a0dfbee88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: HEALTHY } } }
I20260929 02:01:05.933425 17462 tablet_bootstrap.cc:492] T 7f3e8d6962a940868e1ab2caa707cff1 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:05.933593 17462 ts_tablet_manager.cc:1449] T 7f3e8d6962a940868e1ab2caa707cff1 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.000s
I20260929 02:01:05.934082 17398 raft_consensus.cc:493] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:05.934279 17398 raft_consensus.cc:515] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.934530 17462 raft_consensus.cc:359] T 7f3e8d6962a940868e1ab2caa707cff1 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.934732 17462 raft_consensus.cc:385] T 7f3e8d6962a940868e1ab2caa707cff1 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.934819 17462 raft_consensus.cc:740] T 7f3e8d6962a940868e1ab2caa707cff1 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.935019 17398 leader_election.cc:290] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:05.935089 17462 consensus_queue.cc:260] T 7f3e8d6962a940868e1ab2caa707cff1 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.935725 17462 ts_tablet_manager.cc:1480] T 7f3e8d6962a940868e1ab2caa707cff1 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:01:05.935685 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f3e8d6962a940868e1ab2caa707cff1" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:05.935927 16986 raft_consensus.cc:2468] T 7f3e8d6962a940868e1ab2caa707cff1 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:05.936035 17462 tablet_bootstrap.cc:492] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:05.936385 17056 leader_election.cc:304] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:05.936913 17398 raft_consensus.cc:2804] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:05.937014 17398 raft_consensus.cc:493] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:05.937109 17398 raft_consensus.cc:3060] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.937902 17462 tablet_bootstrap.cc:654] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.940156 17398 raft_consensus.cc:515] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.940572 17475 raft_consensus.cc:493] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:05.940742 17475 raft_consensus.cc:515] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.941445 17475 leader_election.cc:290] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:05.942008 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00f69d45340f4d578a7f3b810c6f9fec" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:05.942198 16986 raft_consensus.cc:2468] T 00f69d45340f4d578a7f3b810c6f9fec P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:05.942656 17056 leader_election.cc:304] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:05.944365 17475 raft_consensus.cc:2804] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:05.944482 17475 raft_consensus.cc:493] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:05.944584 17475 raft_consensus.cc:3060] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.945786 17481 tablet_bootstrap.cc:492] T 2e9a65b8f4be4f1d99eef1517d20743c P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:05.945958 17481 ts_tablet_manager.cc:1449] T 2e9a65b8f4be4f1d99eef1517d20743c P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.017s user 0.001s sys 0.003s
I20260929 02:01:05.946732 17481 raft_consensus.cc:359] T 2e9a65b8f4be4f1d99eef1517d20743c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.946935 17481 raft_consensus.cc:385] T 2e9a65b8f4be4f1d99eef1517d20743c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.947000 17481 raft_consensus.cc:740] T 2e9a65b8f4be4f1d99eef1517d20743c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.947228 17481 consensus_queue.cc:260] T 2e9a65b8f4be4f1d99eef1517d20743c P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.947721 17481 ts_tablet_manager.cc:1480] T 2e9a65b8f4be4f1d99eef1517d20743c P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 02:01:05.948026 17481 tablet_bootstrap.cc:492] T 7f3e8d6962a940868e1ab2caa707cff1 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:05.948962 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f3e8d6962a940868e1ab2caa707cff1" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:05.949296 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00f69d45340f4d578a7f3b810c6f9fec" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:05.949461 17251 raft_consensus.cc:2468] T 00f69d45340f4d578a7f3b810c6f9fec P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
W20260929 02:01:05.951057 17055 leader_election.cc:343] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:05.950773 17481 tablet_bootstrap.cc:654] T 7f3e8d6962a940868e1ab2caa707cff1 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.953289 17475 raft_consensus.cc:515] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.953979 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.955178 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f3e8d6962a940868e1ab2caa707cff1" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:05.955478 16986 raft_consensus.cc:3060] T 7f3e8d6962a940868e1ab2caa707cff1 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.957713 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.023s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":22993}
I20260929 02:01:05.958570 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.024s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":23774}
I20260929 02:01:05.958979 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.024s user 0.006s sys 0.000s Metrics: {"bytes_written":466,"delete_count":0,"lbm_write_time_us":178,"lbm_writes_lt_1ms":4,"mutex_wait_us":16808,"reinsert_count":0,"update_count":2}
I20260929 02:01:05.959486 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.047s user 0.010s sys 0.001s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":12221,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":86,"dirs.run_cpu_time_us":230,"dirs.run_wall_time_us":3793,"lbm_read_time_us":4893,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":26,"lbm_write_time_us":297,"lbm_writes_lt_1ms":9,"mutex_wait_us":11006,"peak_mem_usage":32768,"reinsert_count":0,"update_count":6}
I20260929 02:01:05.960690 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f3e8d6962a940868e1ab2caa707cff1" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
W20260929 02:01:05.961326 17055 leader_election.cc:343] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:05.961722 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.962085 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:05.960696 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.049s user 0.011s sys 0.005s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":12221,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":799,"dirs.run_cpu_time_us":247,"dirs.run_wall_time_us":1073,"lbm_read_time_us":516,"lbm_reads_lt_1ms":27,"lbm_write_time_us":320,"lbm_writes_lt_1ms":9,"mutex_wait_us":1705,"peak_mem_usage":32768,"reinsert_count":0,"thread_start_us":15065,"threads_started":1,"update_count":6}
I20260929 02:01:05.962304 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.962407 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:05.962497 17462 tablet_bootstrap.cc:492] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:05.962638 17462 ts_tablet_manager.cc:1449] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.027s user 0.010s sys 0.000s
I20260929 02:01:05.963163 17481 tablet_bootstrap.cc:492] T 7f3e8d6962a940868e1ab2caa707cff1 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:05.963289 17481 ts_tablet_manager.cc:1449] T 7f3e8d6962a940868e1ab2caa707cff1 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.015s user 0.001s sys 0.006s
I20260929 02:01:05.963562 17462 raft_consensus.cc:359] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.963724 17462 raft_consensus.cc:385] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.963812 17462 raft_consensus.cc:740] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.964047 17462 consensus_queue.cc:260] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.964144 17481 raft_consensus.cc:359] T 7f3e8d6962a940868e1ab2caa707cff1 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.961500 17398 leader_election.cc:290] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:05.964310 17481 raft_consensus.cc:385] T 7f3e8d6962a940868e1ab2caa707cff1 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.964684 17462 ts_tablet_manager.cc:1480] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.002s user 0.002s sys 0.001s
I20260929 02:01:05.965252 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00f69d45340f4d578a7f3b810c6f9fec" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:05.965359 17462 tablet_bootstrap.cc:492] T a3f4dfb792b946728e051ae1791cb75a P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:05.965438 16983 raft_consensus.cc:3060] T 00f69d45340f4d578a7f3b810c6f9fec P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.967384 17481 raft_consensus.cc:740] T 7f3e8d6962a940868e1ab2caa707cff1 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.967579 17462 tablet_bootstrap.cc:654] T a3f4dfb792b946728e051ae1791cb75a P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.969446 17481 consensus_queue.cc:260] T 7f3e8d6962a940868e1ab2caa707cff1 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.970049 16983 raft_consensus.cc:2468] T 00f69d45340f4d578a7f3b810c6f9fec P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:05.970532 17056 leader_election.cc:304] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:05.970822 17481 ts_tablet_manager.cc:1480] T 7f3e8d6962a940868e1ab2caa707cff1 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20260929 02:01:05.971119 17481 tablet_bootstrap.cc:492] T 85b63af0846742c99534e9bd449181d3 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:05.974020 17481 tablet_bootstrap.cc:654] T 85b63af0846742c99534e9bd449181d3 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:05.974764 16986 raft_consensus.cc:2468] T 7f3e8d6962a940868e1ab2caa707cff1 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:05.975006 17398 raft_consensus.cc:2804] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:05.978439 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.016s user 0.003s sys 0.001s Metrics: {"bytes_written":466,"delete_count":0,"lbm_write_time_us":154,"lbm_writes_lt_1ms":4,"reinsert_count":0,"spinlock_wait_cycles":1920,"update_count":2}
I20260929 02:01:05.978219 17462 tablet_bootstrap.cc:492] T a3f4dfb792b946728e051ae1791cb75a P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:05.978977 17462 ts_tablet_manager.cc:1449] T a3f4dfb792b946728e051ae1791cb75a P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.014s user 0.003s sys 0.006s
I20260929 02:01:05.979717 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "00f69d45340f4d578a7f3b810c6f9fec" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:05.979909 17251 raft_consensus.cc:3060] T 00f69d45340f4d578a7f3b810c6f9fec P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.980196 17462 raft_consensus.cc:359] T a3f4dfb792b946728e051ae1791cb75a P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.980803 17462 raft_consensus.cc:385] T a3f4dfb792b946728e051ae1791cb75a P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:05.980906 17462 raft_consensus.cc:740] T a3f4dfb792b946728e051ae1791cb75a P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:05.981248 17462 consensus_queue.cc:260] T a3f4dfb792b946728e051ae1791cb75a P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.982013 17462 ts_tablet_manager.cc:1480] T a3f4dfb792b946728e051ae1791cb75a P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260929 02:01:05.983203 17251 raft_consensus.cc:2468] T 00f69d45340f4d578a7f3b810c6f9fec P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:05.984288 17369 raft_consensus.cc:493] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:05.984860 17369 raft_consensus.cc:515] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.985598 17369 leader_election.cc:290] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:05.986644 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2486911d1a704e159ed5ad88ebdd82c1" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:05.986850 16983 raft_consensus.cc:2468] T 2486911d1a704e159ed5ad88ebdd82c1 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:05.987107 17410 consensus_queue.cc:1048] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:05.987474 17056 leader_election.cc:304] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:05.988104 17369 raft_consensus.cc:2804] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:05.988229 17369 raft_consensus.cc:493] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:05.988353 17369 raft_consensus.cc:3060] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.991457 17369 raft_consensus.cc:515] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.992304 17369 leader_election.cc:290] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:05.994021 17410 consensus_queue.cc:1048] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:05.994320 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2486911d1a704e159ed5ad88ebdd82c1" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:05.994318 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2486911d1a704e159ed5ad88ebdd82c1" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:05.994578 17251 raft_consensus.cc:2468] T 2486911d1a704e159ed5ad88ebdd82c1 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:05.994892 17398 raft_consensus.cc:697] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Becoming Leader. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Running, Role: LEADER
I20260929 02:01:05.995190 17398 consensus_queue.cc:237] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:05.994128 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2486911d1a704e159ed5ad88ebdd82c1" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:05.996575 17475 leader_election.cc:290] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:05.996732 17056 leader_election.cc:304] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters: d103dd4823044faebe46100a0dfbee88
I20260929 02:01:05.996773 16983 raft_consensus.cc:3060] T 2486911d1a704e159ed5ad88ebdd82c1 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:05.997097 17475 raft_consensus.cc:2804] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:05.997308 17475 raft_consensus.cc:697] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Becoming Leader. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Running, Role: LEADER
I20260929 02:01:05.997635 17475 consensus_queue.cc:237] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.001600 16983 raft_consensus.cc:2468] T 2486911d1a704e159ed5ad88ebdd82c1 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:06.002490 17056 leader_election.cc:304] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters: d103dd4823044faebe46100a0dfbee88
I20260929 02:01:06.003217 17369 raft_consensus.cc:2804] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:06.003444 17369 raft_consensus.cc:697] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Becoming Leader. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Running, Role: LEADER
I20260929 02:01:06.003856 17369 consensus_queue.cc:237] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.005411 17481 tablet_bootstrap.cc:492] T 85b63af0846742c99534e9bd449181d3 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:06.005661 17481 ts_tablet_manager.cc:1449] T 85b63af0846742c99534e9bd449181d3 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.035s user 0.003s sys 0.024s
I20260929 02:01:06.006779 17481 raft_consensus.cc:359] T 85b63af0846742c99534e9bd449181d3 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.007040 17481 raft_consensus.cc:385] T 85b63af0846742c99534e9bd449181d3 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:06.007135 17481 raft_consensus.cc:740] T 85b63af0846742c99534e9bd449181d3 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:06.007443 17481 consensus_queue.cc:260] T 85b63af0846742c99534e9bd449181d3 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.008153 17481 ts_tablet_manager.cc:1480] T 85b63af0846742c99534e9bd449181d3 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260929 02:01:06.008565 17481 tablet_bootstrap.cc:492] T a4083e6b1e594ee88efbd7e1f04f70e0 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:06.011658 17481 tablet_bootstrap.cc:654] T a4083e6b1e594ee88efbd7e1f04f70e0 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:06.012331 17543 raft_consensus.cc:493] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:06.012568 17543 raft_consensus.cc:515] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.013657 17543 leader_election.cc:290] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:06.014843 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4083e6b1e594ee88efbd7e1f04f70e0" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
W20260929 02:01:06.015625 17055 leader_election.cc:343] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:06.016063 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4083e6b1e594ee88efbd7e1f04f70e0" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:06.016394 16983 raft_consensus.cc:2468] T a4083e6b1e594ee88efbd7e1f04f70e0 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:06.017107 17056 leader_election.cc:304] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters: d103dd4823044faebe46100a0dfbee88
I20260929 02:01:06.017426 17543 raft_consensus.cc:2804] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:06.017547 17543 raft_consensus.cc:493] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:06.017656 17543 raft_consensus.cc:3060] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:06.021716 17543 raft_consensus.cc:515] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.021840 16817 catalog_manager.cc:5838] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66). New cstate: current_term: 1 leader_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:06.022943 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4083e6b1e594ee88efbd7e1f04f70e0" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:06.023221 16983 raft_consensus.cc:3060] T a4083e6b1e594ee88efbd7e1f04f70e0 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:06.026024 16983 raft_consensus.cc:2468] T a4083e6b1e594ee88efbd7e1f04f70e0 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:06.026405 17543 leader_election.cc:290] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:06.027000 17056 leader_election.cc:304] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:06.027376 17543 raft_consensus.cc:2804] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:06.027796 17543 raft_consensus.cc:697] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Becoming Leader. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Running, Role: LEADER
I20260929 02:01:06.028175 17543 consensus_queue.cc:237] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.032002 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.032338 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.032414 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.032667 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.032783 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.032980 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.033051 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.033334 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.033979 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4083e6b1e594ee88efbd7e1f04f70e0" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:06.034862 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.034901 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.035132 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.035152 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
W20260929 02:01:06.035146 17055 leader_election.cc:343] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:06.035352 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.035372 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.035907 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:06.035944 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:06.036186 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:06.036186 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:06.036427 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:06.036427 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:06.036748 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.036741 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.036964 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.037070 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.052593 17480 raft_consensus.cc:493] T 2e9a65b8f4be4f1d99eef1517d20743c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:06.052865 17480 raft_consensus.cc:515] T 2e9a65b8f4be4f1d99eef1517d20743c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.053781 17480 leader_election.cc:290] T 2e9a65b8f4be4f1d99eef1517d20743c P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:06.054327 17119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e9a65b8f4be4f1d99eef1517d20743c" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:06.054584 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e9a65b8f4be4f1d99eef1517d20743c" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:06.054700 17119 raft_consensus.cc:2393] T 2e9a65b8f4be4f1d99eef1517d20743c P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d103dd4823044faebe46100a0dfbee88 in current term 1: Already voted for candidate 97df55a0361841319adfae27c9787b2f in this term.
I20260929 02:01:06.055526 17186 leader_election.cc:304] T 2e9a65b8f4be4f1d99eef1517d20743c P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f
I20260929 02:01:06.056195 17480 raft_consensus.cc:3060] T 2e9a65b8f4be4f1d99eef1517d20743c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:06.060365 17480 raft_consensus.cc:2749] T 2e9a65b8f4be4f1d99eef1517d20743c P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:06.064096 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.031s user 0.005s sys 0.001s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":148,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:06.064306 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.028s user 0.008s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":700,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":69,"lbm_read_time_us":231,"lbm_reads_lt_1ms":8,"lbm_write_time_us":241,"lbm_writes_lt_1ms":4,"mutex_wait_us":18875,"reinsert_count":0,"update_count":3}
I20260929 02:01:06.064931 17481 tablet_bootstrap.cc:492] T a4083e6b1e594ee88efbd7e1f04f70e0 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:06.065109 17481 ts_tablet_manager.cc:1449] T a4083e6b1e594ee88efbd7e1f04f70e0 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.057s user 0.024s sys 0.023s
I20260929 02:01:06.065907 17481 raft_consensus.cc:359] T a4083e6b1e594ee88efbd7e1f04f70e0 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.066077 17481 raft_consensus.cc:385] T a4083e6b1e594ee88efbd7e1f04f70e0 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:06.066191 17481 raft_consensus.cc:740] T a4083e6b1e594ee88efbd7e1f04f70e0 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:06.066422 17481 consensus_queue.cc:260] T a4083e6b1e594ee88efbd7e1f04f70e0 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.066962 17481 ts_tablet_manager.cc:1480] T a4083e6b1e594ee88efbd7e1f04f70e0 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 02:01:06.067158 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.034s user 0.001s sys 0.005s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":175,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:06.067102 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.030s user 0.006s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":700,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"lbm_read_time_us":200,"lbm_reads_lt_1ms":8,"lbm_write_time_us":196,"lbm_writes_lt_1ms":4,"mutex_wait_us":12411,"reinsert_count":0,"update_count":3}
I20260929 02:01:06.068099 17481 tablet_bootstrap.cc:492] T d5da16034f2244a2b335f89f6ab7caa9 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:06.069963 17481 tablet_bootstrap.cc:654] T d5da16034f2244a2b335f89f6ab7caa9 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:06.071671 16817 catalog_manager.cc:5838] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66). New cstate: current_term: 1 leader_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:06.072222 16817 catalog_manager.cc:5838] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66). New cstate: current_term: 1 leader_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:06.072781 16817 catalog_manager.cc:5838] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66). New cstate: current_term: 1 leader_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:06.073354 17481 tablet_bootstrap.cc:492] T d5da16034f2244a2b335f89f6ab7caa9 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:06.073535 17481 ts_tablet_manager.cc:1449] T d5da16034f2244a2b335f89f6ab7caa9 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.005s user 0.005s sys 0.000s
I20260929 02:01:06.074131 17481 raft_consensus.cc:359] T d5da16034f2244a2b335f89f6ab7caa9 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.074316 17481 raft_consensus.cc:385] T d5da16034f2244a2b335f89f6ab7caa9 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:06.074400 17481 raft_consensus.cc:740] T d5da16034f2244a2b335f89f6ab7caa9 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:06.074632 17481 consensus_queue.cc:260] T d5da16034f2244a2b335f89f6ab7caa9 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.075162 17481 ts_tablet_manager.cc:1480] T d5da16034f2244a2b335f89f6ab7caa9 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:01:06.075444 17481 tablet_bootstrap.cc:492] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:06.077134 17481 tablet_bootstrap.cc:654] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:06.082574 17481 tablet_bootstrap.cc:492] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:06.082728 17481 ts_tablet_manager.cc:1449] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.007s user 0.004s sys 0.002s
I20260929 02:01:06.083532 17481 raft_consensus.cc:359] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.083721 17481 raft_consensus.cc:385] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:06.083844 17481 raft_consensus.cc:740] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:06.084086 17481 consensus_queue.cc:260] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.084664 17481 ts_tablet_manager.cc:1480] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.002s user 0.000s sys 0.003s
I20260929 02:01:06.084995 17481 tablet_bootstrap.cc:492] T ea01c17d8d4b405ab885e7eec91b5cbc P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:06.085469 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.086820 17481 tablet_bootstrap.cc:654] T ea01c17d8d4b405ab885e7eec91b5cbc P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:06.088521 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.088855 17481 tablet_bootstrap.cc:492] T ea01c17d8d4b405ab885e7eec91b5cbc P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:06.088865 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.089023 17481 ts_tablet_manager.cc:1449] T ea01c17d8d4b405ab885e7eec91b5cbc P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.004s user 0.001s sys 0.002s
I20260929 02:01:06.089351 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.089710 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.170296
I20260929 02:01:06.089800 17481 raft_consensus.cc:359] T ea01c17d8d4b405ab885e7eec91b5cbc P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.090009 17481 raft_consensus.cc:385] T ea01c17d8d4b405ab885e7eec91b5cbc P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:06.090006 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.170296
I20260929 02:01:06.090083 17481 raft_consensus.cc:740] T ea01c17d8d4b405ab885e7eec91b5cbc P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:06.090286 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.170296
I20260929 02:01:06.090322 17481 consensus_queue.cc:260] T ea01c17d8d4b405ab885e7eec91b5cbc P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.090570 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.090590 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.090616 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.003s sys 0.000s Metrics: {"bytes_written":469,"delete_count":0,"lbm_write_time_us":151,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":3}
I20260929 02:01:06.090920 17481 ts_tablet_manager.cc:1480] T ea01c17d8d4b405ab885e7eec91b5cbc P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:01:06.091709 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:06.091989 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:06.092242 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.094359 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.001s sys 0.002s Metrics: {"bytes_written":469,"delete_count":0,"lbm_write_time_us":138,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":3}
I20260929 02:01:06.099187 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.004s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":760,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":68,"lbm_read_time_us":105,"lbm_reads_lt_1ms":8,"lbm_write_time_us":155,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":6}
I20260929 02:01:06.104040 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.014s user 0.007s sys 0.000s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":12426,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":177,"lbm_read_time_us":456,"lbm_reads_lt_1ms":27,"lbm_write_time_us":320,"lbm_writes_lt_1ms":9,"mutex_wait_us":3977,"peak_mem_usage":32768,"reinsert_count":0,"update_count":6}
I20260929 02:01:06.136538 17480 raft_consensus.cc:493] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:06.136777 17480 raft_consensus.cc:515] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.137610 17480 leader_election.cc:290] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:06.138134 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cd30e2805234d9fadc0d2ad839a6bc3" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:06.138208 17119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cd30e2805234d9fadc0d2ad839a6bc3" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:06.138435 17119 raft_consensus.cc:2468] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 0.
I20260929 02:01:06.138435 16983 raft_consensus.cc:2468] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 0.
I20260929 02:01:06.138945 17186 leader_election.cc:304] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 3 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:06.139230 17480 raft_consensus.cc:2804] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:06.139341 17480 raft_consensus.cc:493] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:06.139441 17480 raft_consensus.cc:3060] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:06.141464 17480 raft_consensus.cc:515] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.142055 17480 leader_election.cc:290] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:06.142452 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cd30e2805234d9fadc0d2ad839a6bc3" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:06.142594 17119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8cd30e2805234d9fadc0d2ad839a6bc3" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:06.142669 16983 raft_consensus.cc:3060] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:06.142813 17119 raft_consensus.cc:3060] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:06.145815 16983 raft_consensus.cc:2468] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 1.
I20260929 02:01:06.146392 17189 leader_election.cc:304] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:06.146838 17480 raft_consensus.cc:2804] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:06.147027 17119 raft_consensus.cc:2468] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 1.
I20260929 02:01:06.147146 17480 raft_consensus.cc:697] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Becoming Leader. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Running, Role: LEADER
I20260929 02:01:06.147529 17480 consensus_queue.cc:237] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.148258 17410 raft_consensus.cc:493] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:06.148412 17410 raft_consensus.cc:515] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.149183 17410 leader_election.cc:290] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:06.149609 17119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b63af0846742c99534e9bd449181d3" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:06.149835 17119 raft_consensus.cc:2468] T 85b63af0846742c99534e9bd449181d3 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:06.150251 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b63af0846742c99534e9bd449181d3" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:06.150297 16919 leader_election.cc:304] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:06.150444 17253 raft_consensus.cc:2468] T 85b63af0846742c99534e9bd449181d3 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:06.150573 17410 raft_consensus.cc:2804] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:06.150717 17410 raft_consensus.cc:493] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:06.150820 17410 raft_consensus.cc:3060] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:06.153664 17410 raft_consensus.cc:515] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.154655 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b63af0846742c99534e9bd449181d3" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:06.154737 17119 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "85b63af0846742c99534e9bd449181d3" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:06.154830 17253 raft_consensus.cc:3060] T 85b63af0846742c99534e9bd449181d3 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:06.154940 17119 raft_consensus.cc:3060] T 85b63af0846742c99534e9bd449181d3 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:06.154724 16818 catalog_manager.cc:5838] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88 reported cstate change: term changed from 0 to 1, leader changed from <none> to d103dd4823044faebe46100a0dfbee88 (127.0.71.67). New cstate: current_term: 1 leader_uuid: "d103dd4823044faebe46100a0dfbee88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: HEALTHY } } }
I20260929 02:01:06.155834 17410 leader_election.cc:290] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Requested vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:06.158752 17253 raft_consensus.cc:2468] T 85b63af0846742c99534e9bd449181d3 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:06.159215 16921 leader_election.cc:304] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:06.159514 17410 raft_consensus.cc:2804] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:06.159665 17410 raft_consensus.cc:697] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Becoming Leader. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Running, Role: LEADER
I20260929 02:01:06.159926 17410 consensus_queue.cc:237] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.167722 17119 raft_consensus.cc:2468] T 85b63af0846742c99534e9bd449181d3 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:06.170914 17480 consensus_queue.cc:1048] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:06.172089 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.172427 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.172897 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.173219 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.173517 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.173822 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.174124 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.174595 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:06.177358 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.177670 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.177937 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.178187 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.180063 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.180339 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.180603 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.182526 17337 consensus_queue.cc:1048] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:06.184203 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.001s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":179,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:06.183179 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.010s user 0.004s sys 0.002s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":197,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:06.188468 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.013s user 0.004s sys 0.004s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":793,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":486,"lbm_read_time_us":150,"lbm_reads_lt_1ms":8,"lbm_write_time_us":141,"lbm_writes_lt_1ms":4,"mutex_wait_us":5602,"reinsert_count":0,"update_count":7}
I20260929 02:01:06.192639 17119 raft_consensus.cc:3060] T 5e7d7ccf81494fec9e066824c69abd61 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:06.198774 17337 consensus_queue.cc:1048] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:06.205068 17382 consensus_queue.cc:1048] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:06.210214 16818 catalog_manager.cc:5838] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97df55a0361841319adfae27c9787b2f (127.0.71.65). New cstate: current_term: 1 leader_uuid: "97df55a0361841319adfae27c9787b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:06.210544 16983 raft_consensus.cc:3060] T a3f4dfb792b946728e051ae1791cb75a P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:06.219472 17480 consensus_queue.cc:1048] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:06.220036 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.220371 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.220726 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.220947 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.221581 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.222167 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.222587 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.102011
I20260929 02:01:06.222622 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.223073 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.102011
I20260929 02:01:06.223366 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:06.224035 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.225875 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.226159 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.226374 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.226650 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.228525 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.228828 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.229069 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.223332 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.232489 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:06.232797 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:06.233042 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:06.233331 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.233580 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.235014 17410 consensus_queue.cc:1048] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:06.239754 17480 consensus_queue.cc:1048] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:06.243568 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.022s user 0.003s sys 0.002s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":127,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:06.244117 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.017s user 0.003s sys 0.002s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":178,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:06.246373 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.013s user 0.006s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":670,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":2711,"lbm_read_time_us":134,"lbm_reads_lt_1ms":8,"lbm_write_time_us":149,"lbm_writes_lt_1ms":4,"mutex_wait_us":3023,"reinsert_count":0,"update_count":2}
I20260929 02:01:06.250218 17337 consensus_queue.cc:1048] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:06.251976 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.029s user 0.011s sys 0.000s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":12486,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":87,"lbm_read_time_us":561,"lbm_reads_lt_1ms":27,"lbm_write_time_us":324,"lbm_writes_lt_1ms":9,"mutex_wait_us":3821,"peak_mem_usage":32768,"reinsert_count":0,"update_count":8}
I20260929 02:01:06.262349 17480 consensus_queue.cc:1048] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:06.272694 17348 consensus_queue.cc:1048] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:06.288165 17348 raft_consensus.cc:493] T 7f3e8d6962a940868e1ab2caa707cff1 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:06.288373 17348 raft_consensus.cc:515] T 7f3e8d6962a940868e1ab2caa707cff1 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.288995 17348 leader_election.cc:290] T 7f3e8d6962a940868e1ab2caa707cff1 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:06.289780 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f3e8d6962a940868e1ab2caa707cff1" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:06.289919 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7f3e8d6962a940868e1ab2caa707cff1" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:06.290618 16983 raft_consensus.cc:2393] T 7f3e8d6962a940868e1ab2caa707cff1 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d103dd4823044faebe46100a0dfbee88 in current term 1: Already voted for candidate 8668afa5b5ca4ebdabc812d0789de7bc in this term.
I20260929 02:01:06.291045 17189 leader_election.cc:304] T 7f3e8d6962a940868e1ab2caa707cff1 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f
I20260929 02:01:06.291340 17348 raft_consensus.cc:3060] T 7f3e8d6962a940868e1ab2caa707cff1 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:06.294427 17348 raft_consensus.cc:2749] T 7f3e8d6962a940868e1ab2caa707cff1 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:06.300252 17561 raft_consensus.cc:493] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:06.300472 17561 raft_consensus.cc:515] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.301216 17561 leader_election.cc:290] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:06.301620 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5da16034f2244a2b335f89f6ab7caa9" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:06.301831 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5da16034f2244a2b335f89f6ab7caa9" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:06.301887 17117 raft_consensus.cc:2468] T d5da16034f2244a2b335f89f6ab7caa9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:06.302048 17253 raft_consensus.cc:2468] T d5da16034f2244a2b335f89f6ab7caa9 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:06.302371 16919 leader_election.cc:304] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:06.302747 17561 raft_consensus.cc:2804] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:06.302862 17561 raft_consensus.cc:493] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:06.302951 17561 raft_consensus.cc:3060] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:06.306118 17561 raft_consensus.cc:515] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.306775 17561 leader_election.cc:290] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Requested vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:06.307299 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5da16034f2244a2b335f89f6ab7caa9" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:06.307345 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d5da16034f2244a2b335f89f6ab7caa9" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:06.307535 17117 raft_consensus.cc:3060] T d5da16034f2244a2b335f89f6ab7caa9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:06.307569 17253 raft_consensus.cc:3060] T d5da16034f2244a2b335f89f6ab7caa9 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:06.310616 17117 raft_consensus.cc:2468] T d5da16034f2244a2b335f89f6ab7caa9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:06.310616 17253 raft_consensus.cc:2468] T d5da16034f2244a2b335f89f6ab7caa9 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:06.311082 16919 leader_election.cc:304] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:06.311599 17561 raft_consensus.cc:2804] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:06.311802 17561 raft_consensus.cc:697] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Becoming Leader. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Running, Role: LEADER
I20260929 02:01:06.312104 17561 consensus_queue.cc:237] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.315075 16818 catalog_manager.cc:5838] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97df55a0361841319adfae27c9787b2f (127.0.71.65). New cstate: current_term: 1 leader_uuid: "97df55a0361841319adfae27c9787b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:06.324883 17253 raft_consensus.cc:3060] T 3f0398c024bf4cc48ec1670fc847c7ab P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:06.326617 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.326925 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.327276 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.328102 17475 consensus_queue.cc:1048] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:06.329814 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.330130 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.330411 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.330657 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.330924 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.331073 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.330935 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.331489 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:06.331745 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:06.331748 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.002s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":184,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:06.332168 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.339062 17398 consensus_queue.cc:1048] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:06.339690 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.009s user 0.003s sys 0.002s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":163,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:06.345729 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.014s user 0.005s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":700,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":58,"lbm_read_time_us":174,"lbm_reads_lt_1ms":8,"lbm_write_time_us":150,"lbm_writes_lt_1ms":4,"mutex_wait_us":7601,"reinsert_count":0,"update_count":3}
I20260929 02:01:06.360299 17348 consensus_queue.cc:1048] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:06.365978 17382 consensus_queue.cc:1048] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:06.371394 17382 raft_consensus.cc:493] T a4083e6b1e594ee88efbd7e1f04f70e0 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:06.371603 17382 raft_consensus.cc:515] T a4083e6b1e594ee88efbd7e1f04f70e0 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.372232 17382 leader_election.cc:290] T a4083e6b1e594ee88efbd7e1f04f70e0 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:06.372557 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4083e6b1e594ee88efbd7e1f04f70e0" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:06.372650 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a4083e6b1e594ee88efbd7e1f04f70e0" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:06.372833 16983 raft_consensus.cc:2393] T a4083e6b1e594ee88efbd7e1f04f70e0 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d103dd4823044faebe46100a0dfbee88 in current term 1: Already voted for candidate 8668afa5b5ca4ebdabc812d0789de7bc in this term.
I20260929 02:01:06.373297 17189 leader_election.cc:304] T a4083e6b1e594ee88efbd7e1f04f70e0 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f
I20260929 02:01:06.373564 17382 raft_consensus.cc:3060] T a4083e6b1e594ee88efbd7e1f04f70e0 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:06.375532 17382 raft_consensus.cc:2749] T a4083e6b1e594ee88efbd7e1f04f70e0 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:06.392041 17253 raft_consensus.cc:3060] T ea01c17d8d4b405ab885e7eec91b5cbc P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:06.392671 17410 consensus_queue.cc:1048] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:06.395167 17410 consensus_queue.cc:1048] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:06.399997 17398 consensus_queue.cc:1048] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:06.408828 17369 consensus_queue.cc:1048] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:06.426497 17369 consensus_queue.cc:1048] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:06.431813 17475 consensus_queue.cc:1048] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:06.475251 17475 consensus_queue.cc:1048] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:06.481531 17398 consensus_queue.cc:1048] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:06.531888 16819 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44708:
table { table_name: "table-1" } modify_external_catalogs: true
I20260929 02:01:06.532140 16819 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44708:
table { table_name: "table-1" } modify_external_catalogs: true
I20260929 02:01:06.542155 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.542661 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.544610 16819 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet ea01c17d8d4b405ab885e7eec91b5cbc
I20260929 02:01:06.545460 16953 tablet_service.cc:1558] Processing DeleteTablet for tablet ea01c17d8d4b405ab885e7eec91b5cbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:06.545917 17094 tablet_service.cc:1558] Processing DeleteTablet for tablet ea01c17d8d4b405ab885e7eec91b5cbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:06.546321 17228 tablet_service.cc:1558] Processing DeleteTablet for tablet ea01c17d8d4b405ab885e7eec91b5cbc with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:06.549360 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.549746 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.550037 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.550351 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.545704 16819 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 20a08b5cc6ae46ebb4696c9f3c83c5fa
I20260929 02:01:06.552805 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.552999 16955 tablet_service.cc:1558] Processing DeleteTablet for tablet 20a08b5cc6ae46ebb4696c9f3c83c5fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:06.553128 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.553359 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.553568 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.553872 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.555229 17088 tablet_service.cc:1558] Processing DeleteTablet for tablet 20a08b5cc6ae46ebb4696c9f3c83c5fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:06.556255 16819 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 9603392111a04ed395f159d69ac4801c
I20260929 02:01:06.556847 17226 tablet_service.cc:1558] Processing DeleteTablet for tablet 20a08b5cc6ae46ebb4696c9f3c83c5fa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:06.556994 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.604364
I20260929 02:01:06.557173 17584 tablet_replica.cc:333] T ea01c17d8d4b405ab885e7eec91b5cbc P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:06.557309 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.604364
I20260929 02:01:06.557338 16955 tablet_service.cc:1558] Processing DeleteTablet for tablet 9603392111a04ed395f159d69ac4801c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:06.557713 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.558082 17583 tablet_replica.cc:333] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:06.558130 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:06.558208 17584 raft_consensus.cc:2243] T ea01c17d8d4b405ab885e7eec91b5cbc P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:06.558234 17094 tablet_service.cc:1558] Processing DeleteTablet for tablet 9603392111a04ed395f159d69ac4801c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:06.558319 17583 raft_consensus.cc:2243] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:06.558451 17584 raft_consensus.cc:2272] T ea01c17d8d4b405ab885e7eec91b5cbc P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.558540 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.558702 17583 raft_consensus.cc:2272] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.558982 16819 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 7822796e1a01411aaa1c2b394cc27263
I20260929 02:01:06.558998 17226 tablet_service.cc:1558] Processing DeleteTablet for tablet 9603392111a04ed395f159d69ac4801c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:06.559618 16953 tablet_service.cc:1558] Processing DeleteTablet for tablet 7822796e1a01411aaa1c2b394cc27263 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:06.559923 17094 tablet_service.cc:1558] Processing DeleteTablet for tablet 7822796e1a01411aaa1c2b394cc27263 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:06.560256 17584 ts_tablet_manager.cc:1962] T ea01c17d8d4b405ab885e7eec91b5cbc P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.560290 16819 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 2486911d1a704e159ed5ad88ebdd82c1
I20260929 02:01:06.560770 17226 tablet_service.cc:1558] Processing DeleteTablet for tablet 7822796e1a01411aaa1c2b394cc27263 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:06.561161 16953 tablet_service.cc:1558] Processing DeleteTablet for tablet 2486911d1a704e159ed5ad88ebdd82c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:06.561424 17094 tablet_service.cc:1558] Processing DeleteTablet for tablet 2486911d1a704e159ed5ad88ebdd82c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:06.561766 16819 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 8cd30e2805234d9fadc0d2ad839a6bc3
I20260929 02:01:06.562165 17226 tablet_service.cc:1558] Processing DeleteTablet for tablet 2486911d1a704e159ed5ad88ebdd82c1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:06.562417 16953 tablet_service.cc:1558] Processing DeleteTablet for tablet 8cd30e2805234d9fadc0d2ad839a6bc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:06.562880 17094 tablet_service.cc:1558] Processing DeleteTablet for tablet 8cd30e2805234d9fadc0d2ad839a6bc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:06.563035 16819 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 3f0398c024bf4cc48ec1670fc847c7ab
I20260929 02:01:06.563431 17226 tablet_service.cc:1558] Processing DeleteTablet for tablet 8cd30e2805234d9fadc0d2ad839a6bc3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:06.563766 16953 tablet_service.cc:1558] Processing DeleteTablet for tablet 3f0398c024bf4cc48ec1670fc847c7ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:06.564117 17094 tablet_service.cc:1558] Processing DeleteTablet for tablet 3f0398c024bf4cc48ec1670fc847c7ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:06.564289 16819 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 2e9a65b8f4be4f1d99eef1517d20743c
I20260929 02:01:06.565013 17226 tablet_service.cc:1558] Processing DeleteTablet for tablet 3f0398c024bf4cc48ec1670fc847c7ab with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:06.565436 17094 tablet_service.cc:1558] Processing DeleteTablet for tablet 2e9a65b8f4be4f1d99eef1517d20743c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:06.565586 16819 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 00f69d45340f4d578a7f3b810c6f9fec
I20260929 02:01:06.565785 17583 ts_tablet_manager.cc:1962] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.566073 17226 tablet_service.cc:1558] Processing DeleteTablet for tablet 2e9a65b8f4be4f1d99eef1517d20743c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:06.566221 16955 tablet_service.cc:1558] Processing DeleteTablet for tablet 00f69d45340f4d578a7f3b810c6f9fec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:06.566402 16953 tablet_service.cc:1558] Processing DeleteTablet for tablet 2e9a65b8f4be4f1d99eef1517d20743c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:06.566663 17585 tablet_replica.cc:333] T ea01c17d8d4b405ab885e7eec91b5cbc P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:06.566872 17585 raft_consensus.cc:2243] T ea01c17d8d4b405ab885e7eec91b5cbc P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:06.567118 17585 raft_consensus.cc:2272] T ea01c17d8d4b405ab885e7eec91b5cbc P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.567894 17094 tablet_service.cc:1558] Processing DeleteTablet for tablet 00f69d45340f4d578a7f3b810c6f9fec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:06.568303 16819 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet a4083e6b1e594ee88efbd7e1f04f70e0
I20260929 02:01:06.568321 17226 tablet_service.cc:1558] Processing DeleteTablet for tablet 00f69d45340f4d578a7f3b810c6f9fec with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:06.568827 17585 ts_tablet_manager.cc:1962] T ea01c17d8d4b405ab885e7eec91b5cbc P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.569046 16953 tablet_service.cc:1558] Processing DeleteTablet for tablet a4083e6b1e594ee88efbd7e1f04f70e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:06.569463 17094 tablet_service.cc:1558] Processing DeleteTablet for tablet a4083e6b1e594ee88efbd7e1f04f70e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:06.569923 17228 tablet_service.cc:1558] Processing DeleteTablet for tablet a4083e6b1e594ee88efbd7e1f04f70e0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:06.570115 16819 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet d5da16034f2244a2b335f89f6ab7caa9
I20260929 02:01:06.571111 17094 tablet_service.cc:1558] Processing DeleteTablet for tablet d5da16034f2244a2b335f89f6ab7caa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:06.571398 16953 tablet_service.cc:1558] Processing DeleteTablet for tablet d5da16034f2244a2b335f89f6ab7caa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:06.571627 17228 tablet_service.cc:1558] Processing DeleteTablet for tablet d5da16034f2244a2b335f89f6ab7caa9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:06.571973 16819 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 85b63af0846742c99534e9bd449181d3
I20260929 02:01:06.573174 17094 tablet_service.cc:1558] Processing DeleteTablet for tablet 85b63af0846742c99534e9bd449181d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:06.573413 16819 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 5e7d7ccf81494fec9e066824c69abd61
I20260929 02:01:06.573601 17228 tablet_service.cc:1558] Processing DeleteTablet for tablet 85b63af0846742c99534e9bd449181d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:06.574879 17094 tablet_service.cc:1558] Processing DeleteTablet for tablet 5e7d7ccf81494fec9e066824c69abd61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:06.574863 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.018s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":18117}
I20260929 02:01:06.575181 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.018s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":18371}
I20260929 02:01:06.575250 17584 ts_tablet_manager.cc:1975] T ea01c17d8d4b405ab885e7eec91b5cbc P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:06.575368 17584 log.cc:1199] T ea01c17d8d4b405ab885e7eec91b5cbc P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/ea01c17d8d4b405ab885e7eec91b5cbc
I20260929 02:01:06.575400 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.032s user 0.005s sys 0.000s Metrics: {"bytes_written":713,"delete_count":0,"lbm_write_time_us":167,"lbm_writes_lt_1ms":4,"mutex_wait_us":25090,"reinsert_count":0,"update_count":12}
I20260929 02:01:06.575829 17584 ts_tablet_manager.cc:1996] T ea01c17d8d4b405ab885e7eec91b5cbc P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:06.575819 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.135877
I20260929 02:01:06.576128 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.135877
I20260929 02:01:06.576402 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.135877
I20260929 02:01:06.576848 17584 tablet_replica.cc:333] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:06.576939 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet ea01c17d8d4b405ab885e7eec91b5cbc (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:06.577064 17584 raft_consensus.cc:2243] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:06.577235 17584 raft_consensus.cc:2272] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.577299 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.572415 17583 ts_tablet_manager.cc:1975] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:06.572628 16953 tablet_service.cc:1558] Processing DeleteTablet for tablet 85b63af0846742c99534e9bd449181d3 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:06.577553 17583 log.cc:1199] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/ea01c17d8d4b405ab885e7eec91b5cbc
I20260929 02:01:06.574331 16955 tablet_service.cc:1558] Processing DeleteTablet for tablet 5e7d7ccf81494fec9e066824c69abd61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:06.577832 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.003000
I20260929 02:01:06.578027 17585 ts_tablet_manager.cc:1975] T ea01c17d8d4b405ab885e7eec91b5cbc P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:06.578042 17583 ts_tablet_manager.cc:1996] T ea01c17d8d4b405ab885e7eec91b5cbc P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:06.578189 17585 log.cc:1199] T ea01c17d8d4b405ab885e7eec91b5cbc P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/ea01c17d8d4b405ab885e7eec91b5cbc
I20260929 02:01:06.578150 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.578619 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.578685 17585 ts_tablet_manager.cc:1996] T ea01c17d8d4b405ab885e7eec91b5cbc P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:06.579083 17583 tablet_replica.cc:333] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:06.579676 17585 tablet_replica.cc:333] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:06.579823 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet ea01c17d8d4b405ab885e7eec91b5cbc (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:06.579921 17585 raft_consensus.cc:2243] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:06.579914 17583 raft_consensus.cc:2243] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:06.580096 17585 raft_consensus.cc:2272] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.580195 17228 tablet_service.cc:1558] Processing DeleteTablet for tablet 5e7d7ccf81494fec9e066824c69abd61 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:06.580240 17583 raft_consensus.cc:2272] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.580410 16819 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 044313d0ac094292808355c152d310aa
I20260929 02:01:06.580693 17584 ts_tablet_manager.cc:1962] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.580772 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet ea01c17d8d4b405ab885e7eec91b5cbc (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:06.581123 17561 consensus_queue.cc:1048] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:06.581254 17094 tablet_service.cc:1558] Processing DeleteTablet for tablet 044313d0ac094292808355c152d310aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:06.581552 17585 ts_tablet_manager.cc:1962] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.581568 17561 consensus_queue.cc:1048] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:06.581722 17228 tablet_service.cc:1558] Processing DeleteTablet for tablet 044313d0ac094292808355c152d310aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:06.581910 16819 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 7f3e8d6962a940868e1ab2caa707cff1
I20260929 02:01:06.582330 16955 tablet_service.cc:1558] Processing DeleteTablet for tablet 044313d0ac094292808355c152d310aa with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:06.582543 16953 tablet_service.cc:1558] Processing DeleteTablet for tablet 7f3e8d6962a940868e1ab2caa707cff1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:06.582827 17094 tablet_service.cc:1558] Processing DeleteTablet for tablet 7f3e8d6962a940868e1ab2caa707cff1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:06.583082 17228 tablet_service.cc:1558] Processing DeleteTablet for tablet 7f3e8d6962a940868e1ab2caa707cff1 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:06.583261 16819 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet a3f4dfb792b946728e051ae1791cb75a
I20260929 02:01:06.583562 17583 ts_tablet_manager.cc:1962] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.584671 16953 tablet_service.cc:1558] Processing DeleteTablet for tablet a3f4dfb792b946728e051ae1791cb75a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:06.586601 17252 raft_consensus.cc:3060] T 2486911d1a704e159ed5ad88ebdd82c1 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:06.588737 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.046s user 0.006s sys 0.000s Metrics: {"bytes_written":628,"delete_count":0,"lbm_write_time_us":187,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":5}
I20260929 02:01:06.589102 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.039s user 0.006s sys 0.000s Metrics: {"bytes_written":917,"delete_count":0,"lbm_write_time_us":186,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":17}
I20260929 02:01:06.590128 17398 consensus_queue.cc:1048] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:06.591921 17094 tablet_service.cc:1558] Processing DeleteTablet for tablet a3f4dfb792b946728e051ae1791cb75a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:06.592968 17228 tablet_service.cc:1558] Processing DeleteTablet for tablet a3f4dfb792b946728e051ae1791cb75a with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:06 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:06.595566 16818 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44708:
name: "table-2"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "col"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 16
seed: 0
}
}
I20260929 02:01:06.596086 17585 ts_tablet_manager.cc:1975] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:06.596211 17585 log.cc:1199] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/20a08b5cc6ae46ebb4696c9f3c83c5fa
W20260929 02:01:06.596531 16818 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table-2 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:01:06.596660 17585 ts_tablet_manager.cc:1996] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:06.597697 17585 tablet_replica.cc:333] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:06.597923 17585 raft_consensus.cc:2243] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:06.598320 17585 raft_consensus.cc:2272] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.599954 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 20a08b5cc6ae46ebb4696c9f3c83c5fa (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:06.601438 17583 ts_tablet_manager.cc:1975] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:06.601635 17583 log.cc:1199] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/20a08b5cc6ae46ebb4696c9f3c83c5fa
I20260929 02:01:06.602104 17583 ts_tablet_manager.cc:1996] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:06.602016 17398 consensus_queue.cc:1048] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:06.603693 17585 ts_tablet_manager.cc:1962] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.604691 17583 tablet_replica.cc:333] T 9603392111a04ed395f159d69ac4801c P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:06.604904 17583 raft_consensus.cc:2243] T 9603392111a04ed395f159d69ac4801c P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:06.605113 17583 raft_consensus.cc:2272] T 9603392111a04ed395f159d69ac4801c P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.606220 17583 ts_tablet_manager.cc:1962] T 9603392111a04ed395f159d69ac4801c P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.607851 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 20a08b5cc6ae46ebb4696c9f3c83c5fa (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:06.610283 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.620229 17585 ts_tablet_manager.cc:1975] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:06.621475 17585 log.cc:1199] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/9603392111a04ed395f159d69ac4801c
I20260929 02:01:06.622073 17585 ts_tablet_manager.cc:1996] T 9603392111a04ed395f159d69ac4801c P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:06.622079 17584 ts_tablet_manager.cc:1975] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:06.622216 17584 log.cc:1199] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/20a08b5cc6ae46ebb4696c9f3c83c5fa
I20260929 02:01:06.622637 17584 ts_tablet_manager.cc:1996] T 20a08b5cc6ae46ebb4696c9f3c83c5fa P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:06.623422 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 9603392111a04ed395f159d69ac4801c (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:06.623560 17584 tablet_replica.cc:333] T 9603392111a04ed395f159d69ac4801c P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:06.623747 17584 raft_consensus.cc:2243] T 9603392111a04ed395f159d69ac4801c P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:06.623751 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 20a08b5cc6ae46ebb4696c9f3c83c5fa (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:06.623904 17585 tablet_replica.cc:333] T 7822796e1a01411aaa1c2b394cc27263 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:06.623968 17584 raft_consensus.cc:2272] T 9603392111a04ed395f159d69ac4801c P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.624189 17585 raft_consensus.cc:2243] T 7822796e1a01411aaa1c2b394cc27263 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:06.624840 17585 raft_consensus.cc:2272] T 7822796e1a01411aaa1c2b394cc27263 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.624863 17584 ts_tablet_manager.cc:1962] T 9603392111a04ed395f159d69ac4801c P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.625742 17585 ts_tablet_manager.cc:1962] T 7822796e1a01411aaa1c2b394cc27263 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.628690 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.071s user 0.013s sys 0.000s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":13113,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":7069,"lbm_read_time_us":545,"lbm_reads_lt_1ms":27,"lbm_write_time_us":312,"lbm_writes_lt_1ms":9,"mutex_wait_us":3451,"peak_mem_usage":32768,"reinsert_count":0,"update_count":18}
I20260929 02:01:06.631745 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.633142 17584 ts_tablet_manager.cc:1975] T 9603392111a04ed395f159d69ac4801c P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:06.633335 17584 log.cc:1199] T 9603392111a04ed395f159d69ac4801c P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/9603392111a04ed395f159d69ac4801c
I20260929 02:01:06.633602 17585 ts_tablet_manager.cc:1975] T 7822796e1a01411aaa1c2b394cc27263 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:06.633821 17584 ts_tablet_manager.cc:1996] T 9603392111a04ed395f159d69ac4801c P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:06.634070 17585 log.cc:1199] T 7822796e1a01411aaa1c2b394cc27263 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/7822796e1a01411aaa1c2b394cc27263
I20260929 02:01:06.634603 17585 ts_tablet_manager.cc:1996] T 7822796e1a01411aaa1c2b394cc27263 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:06.634766 17584 tablet_replica.cc:333] T 7822796e1a01411aaa1c2b394cc27263 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:06.634961 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 9603392111a04ed395f159d69ac4801c (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:06.635031 17584 raft_consensus.cc:2243] T 7822796e1a01411aaa1c2b394cc27263 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:06.635300 17584 raft_consensus.cc:2272] T 7822796e1a01411aaa1c2b394cc27263 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.635691 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 7822796e1a01411aaa1c2b394cc27263 (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:06.636010 17585 tablet_replica.cc:333] T 2486911d1a704e159ed5ad88ebdd82c1 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:06.636250 17585 raft_consensus.cc:2243] T 2486911d1a704e159ed5ad88ebdd82c1 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:06.636283 17584 ts_tablet_manager.cc:1962] T 7822796e1a01411aaa1c2b394cc27263 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.636559 17585 raft_consensus.cc:2272] T 2486911d1a704e159ed5ad88ebdd82c1 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.636977 17583 ts_tablet_manager.cc:1975] T 9603392111a04ed395f159d69ac4801c P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:06.637151 17583 log.cc:1199] T 9603392111a04ed395f159d69ac4801c P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/9603392111a04ed395f159d69ac4801c
I20260929 02:01:06.637519 17585 ts_tablet_manager.cc:1962] T 2486911d1a704e159ed5ad88ebdd82c1 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.637620 17583 ts_tablet_manager.cc:1996] T 9603392111a04ed395f159d69ac4801c P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:06.637844 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.061s user 0.013s sys 0.001s Metrics: {"cfile_cache_miss":9,"cfile_cache_miss_bytes":13412,"cfile_init":6,"delete_count":0,"delta_blocks_compacted":3,"delta_iterators_relevant":3,"dirs.queue_time_us":720,"dirs.run_cpu_time_us":241,"dirs.run_wall_time_us":1083,"lbm_read_time_us":728,"lbm_reads_lt_1ms":33,"lbm_write_time_us":377,"lbm_writes_lt_1ms":9,"peak_mem_usage":32768,"reinsert_count":0,"thread_start_us":7231,"threads_started":1,"update_count":20}
I20260929 02:01:06.639569 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260929 02:01:06.639904 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260929 02:01:06.640223 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260929 02:01:06.640866 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20260929 02:01:06.641234 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.641517 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.642371 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.293531
I20260929 02:01:06.642851 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.293531
I20260929 02:01:06.643190 17584 ts_tablet_manager.cc:1975] T 7822796e1a01411aaa1c2b394cc27263 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:06.643328 17584 log.cc:1199] T 7822796e1a01411aaa1c2b394cc27263 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/7822796e1a01411aaa1c2b394cc27263
I20260929 02:01:06.643420 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.293531
I20260929 02:01:06.643776 17584 ts_tablet_manager.cc:1996] T 7822796e1a01411aaa1c2b394cc27263 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:06.643916 17585 ts_tablet_manager.cc:1975] T 2486911d1a704e159ed5ad88ebdd82c1 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:06.644032 17585 log.cc:1199] T 2486911d1a704e159ed5ad88ebdd82c1 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/2486911d1a704e159ed5ad88ebdd82c1
I20260929 02:01:06.644471 17585 ts_tablet_manager.cc:1996] T 2486911d1a704e159ed5ad88ebdd82c1 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:06.644773 17584 tablet_replica.cc:333] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:06.644972 17584 raft_consensus.cc:2243] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:06.645339 17584 raft_consensus.cc:2272] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.645676 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 2486911d1a704e159ed5ad88ebdd82c1 (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:06.646054 17585 tablet_replica.cc:333] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:06.646155 17584 ts_tablet_manager.cc:1962] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.646258 17585 raft_consensus.cc:2243] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:06.646504 17585 pending_rounds.cc:70] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88: Trying to abort 1 pending ops.
I20260929 02:01:06.646584 17585 pending_rounds.cc:77] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7334491202143027200 op_type: NO_OP noop_request { }
I20260929 02:01:06.646687 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 7822796e1a01411aaa1c2b394cc27263 (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:06.646718 17585 raft_consensus.cc:2889] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260929 02:01:06.646832 17585 raft_consensus.cc:2272] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.647680 17585 ts_tablet_manager.cc:1962] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.653657 17584 ts_tablet_manager.cc:1975] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:06.653921 17584 log.cc:1199] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/2486911d1a704e159ed5ad88ebdd82c1
I20260929 02:01:06.654455 17584 ts_tablet_manager.cc:1996] T 2486911d1a704e159ed5ad88ebdd82c1 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:06.654701 17585 ts_tablet_manager.cc:1975] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:06.654896 17585 log.cc:1199] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/8cd30e2805234d9fadc0d2ad839a6bc3
I20260929 02:01:06.655507 17585 ts_tablet_manager.cc:1996] T 8cd30e2805234d9fadc0d2ad839a6bc3 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:06.656641 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 2486911d1a704e159ed5ad88ebdd82c1 (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:06.656759 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 8cd30e2805234d9fadc0d2ad839a6bc3 (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:06.657130 17584 tablet_replica.cc:333] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:06.657150 17585 tablet_replica.cc:333] T 3f0398c024bf4cc48ec1670fc847c7ab P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:06.657366 17584 raft_consensus.cc:2243] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:06.657423 17585 raft_consensus.cc:2243] T 3f0398c024bf4cc48ec1670fc847c7ab P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:06.657553 17584 raft_consensus.cc:2272] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.657676 17585 raft_consensus.cc:2272] T 3f0398c024bf4cc48ec1670fc847c7ab P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.658349 17584 ts_tablet_manager.cc:1962] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.666360 17584 ts_tablet_manager.cc:1975] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260929 02:01:06.666536 17584 log.cc:1199] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/8cd30e2805234d9fadc0d2ad839a6bc3
I20260929 02:01:06.668673 17584 ts_tablet_manager.cc:1996] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:06.669852 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 8cd30e2805234d9fadc0d2ad839a6bc3 (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:06.670337 17584 tablet_replica.cc:333] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:06.670533 17584 raft_consensus.cc:2243] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:06.670763 17585 ts_tablet_manager.cc:1962] T 3f0398c024bf4cc48ec1670fc847c7ab P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.670920 17584 raft_consensus.cc:2272] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.671829 17584 ts_tablet_manager.cc:1962] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.673329 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 9603392111a04ed395f159d69ac4801c (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:06.674013 17583 tablet_replica.cc:333] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:06.674252 17583 raft_consensus.cc:2243] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:06.674604 17583 raft_consensus.cc:2272] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.677114 17583 ts_tablet_manager.cc:1962] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.679333 17584 ts_tablet_manager.cc:1975] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:06.679477 17584 log.cc:1199] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/3f0398c024bf4cc48ec1670fc847c7ab
I20260929 02:01:06.685017 16806 tablet.cc:2133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260929 02:01:06.689533 16953 tablet_service.cc:1511] Processing CreateTablet for tablet d9511e2734f1414ba6df2e7c9459ce34 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.689836 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.037s user 0.009s sys 0.000s Metrics: {"dirs.queue_time_us":5124,"dirs.run_cpu_time_us":762,"dirs.run_wall_time_us":3522,"lbm_write_time_us":309,"lbm_writes_lt_1ms":7,"num_input_rowsets":0}
I20260929 02:01:06.690546 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.690536 17094 tablet_service.cc:1511] Processing CreateTablet for tablet d9511e2734f1414ba6df2e7c9459ce34 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.691013 17094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9511e2734f1414ba6df2e7c9459ce34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.693682 17584 ts_tablet_manager.cc:1996] T 3f0398c024bf4cc48ec1670fc847c7ab P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:06.693861 17585 ts_tablet_manager.cc:1975] T 3f0398c024bf4cc48ec1670fc847c7ab P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:06.694022 17585 log.cc:1199] T 3f0398c024bf4cc48ec1670fc847c7ab P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/3f0398c024bf4cc48ec1670fc847c7ab
I20260929 02:01:06.695928 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 3f0398c024bf4cc48ec1670fc847c7ab (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:06.696204 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.002s sys 0.000s Metrics: {"bytes_written":682,"delete_count":0,"lbm_write_time_us":231,"lbm_writes_lt_1ms":4,"reinsert_count":0,"spinlock_wait_cycles":6016,"update_count":16}
I20260929 02:01:06.698798 17583 ts_tablet_manager.cc:1975] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:06.698807 16953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9511e2734f1414ba6df2e7c9459ce34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.698979 17583 log.cc:1199] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/7822796e1a01411aaa1c2b394cc27263
I20260929 02:01:06.699457 17583 ts_tablet_manager.cc:1996] T 7822796e1a01411aaa1c2b394cc27263 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:06.700523 17583 tablet_replica.cc:333] T 2486911d1a704e159ed5ad88ebdd82c1 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:06.703042 17584 tablet_replica.cc:333] T 2e9a65b8f4be4f1d99eef1517d20743c P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:06.703255 17584 raft_consensus.cc:2243] T 2e9a65b8f4be4f1d99eef1517d20743c P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:06.703493 17584 raft_consensus.cc:2272] T 2e9a65b8f4be4f1d99eef1517d20743c P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.700724 17583 raft_consensus.cc:2243] T 2486911d1a704e159ed5ad88ebdd82c1 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:06.704336 17583 raft_consensus.cc:2272] T 2486911d1a704e159ed5ad88ebdd82c1 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.704458 17584 ts_tablet_manager.cc:1962] T 2e9a65b8f4be4f1d99eef1517d20743c P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.704934 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 7822796e1a01411aaa1c2b394cc27263 (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:06.705386 17585 ts_tablet_manager.cc:1996] T 3f0398c024bf4cc48ec1670fc847c7ab P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:06.705399 17583 ts_tablet_manager.cc:1962] T 2486911d1a704e159ed5ad88ebdd82c1 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 02:01:06.711591 16919 consensus_peers.cc:597] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f -> Peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Couldn't send request to peer 8668afa5b5ca4ebdabc812d0789de7bc. Error code: TABLET_NOT_FOUND (6). Status: Illegal state: Tablet not RUNNING: STOPPED. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:01:06.711803 17359 consensus_queue.cc:1048] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:06.713784 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 3f0398c024bf4cc48ec1670fc847c7ab (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:06.714184 17585 tablet_replica.cc:333] T 2e9a65b8f4be4f1d99eef1517d20743c P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:06.714457 17585 raft_consensus.cc:2243] T 2e9a65b8f4be4f1d99eef1517d20743c P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:06.714704 17585 raft_consensus.cc:2272] T 2e9a65b8f4be4f1d99eef1517d20743c P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.717056 17585 ts_tablet_manager.cc:1962] T 2e9a65b8f4be4f1d99eef1517d20743c P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.717217 17584 ts_tablet_manager.cc:1975] T 2e9a65b8f4be4f1d99eef1517d20743c P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:06.717398 17584 log.cc:1199] T 2e9a65b8f4be4f1d99eef1517d20743c P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/2e9a65b8f4be4f1d99eef1517d20743c
I20260929 02:01:06.717971 17584 ts_tablet_manager.cc:1996] T 2e9a65b8f4be4f1d99eef1517d20743c P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:06.719303 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 2e9a65b8f4be4f1d99eef1517d20743c (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:06.636174 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.025s user 0.010s sys 0.000s Metrics: {"bytes_written":6941,"cfile_init":1,"dirs.queue_time_us":199,"dirs.run_cpu_time_us":143,"dirs.run_wall_time_us":1109,"drs_written":1,"lbm_read_time_us":89,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1130,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":17}
I20260929 02:01:06.722847 17228 tablet_service.cc:1511] Processing CreateTablet for tablet d9511e2734f1414ba6df2e7c9459ce34 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.723331 17228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d9511e2734f1414ba6df2e7c9459ce34. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.726625 17584 tablet_replica.cc:333] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:06.726904 17584 raft_consensus.cc:2243] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:06.727368 17584 raft_consensus.cc:2272] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.728286 17584 ts_tablet_manager.cc:1962] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.731546 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 2583 bytes on disk
I20260929 02:01:06.731892 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 2583 bytes on disk
I20260929 02:01:06.732975 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4}
I20260929 02:01:06.733574 17337 consensus_queue.cc:1048] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:06.733969 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.737877 16955 tablet_service.cc:1511] Processing CreateTablet for tablet 925a0eec8efb496ba538b7f7c44d84f2 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.738360 16955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 925a0eec8efb496ba538b7f7c44d84f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.750449 16807 tablet.cc:2133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260929 02:01:06.750578 17595 tablet_bootstrap.cc:492] T d9511e2734f1414ba6df2e7c9459ce34 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:06.751106 17088 tablet_service.cc:1511] Processing CreateTablet for tablet 925a0eec8efb496ba538b7f7c44d84f2 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.751765 17088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 925a0eec8efb496ba538b7f7c44d84f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.752199 17584 ts_tablet_manager.cc:1975] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:06.752318 17584 log.cc:1199] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/00f69d45340f4d578a7f3b810c6f9fec
I20260929 02:01:06.752828 17584 ts_tablet_manager.cc:1996] T 00f69d45340f4d578a7f3b810c6f9fec P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:06.753930 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 00f69d45340f4d578a7f3b810c6f9fec (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:06.755236 17583 ts_tablet_manager.cc:1975] T 2486911d1a704e159ed5ad88ebdd82c1 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:06.755386 17583 log.cc:1199] T 2486911d1a704e159ed5ad88ebdd82c1 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/2486911d1a704e159ed5ad88ebdd82c1
I20260929 02:01:06.755402 17097 tablet_service.cc:1511] Processing CreateTablet for tablet 167d937681dc4e509f1d5d4ce2358d27 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.755872 17097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 167d937681dc4e509f1d5d4ce2358d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.755860 17583 ts_tablet_manager.cc:1996] T 2486911d1a704e159ed5ad88ebdd82c1 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:06.756819 17584 tablet_replica.cc:333] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:06.757416 17584 raft_consensus.cc:2243] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:06.757507 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 2486911d1a704e159ed5ad88ebdd82c1 (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:06.757773 17584 raft_consensus.cc:2272] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.757975 17592 tablet_bootstrap.cc:492] T d9511e2734f1414ba6df2e7c9459ce34 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:06.758355 17583 tablet_replica.cc:333] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:06.758648 17583 raft_consensus.cc:2243] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:06.758896 17583 raft_consensus.cc:2272] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.759771 17583 ts_tablet_manager.cc:1962] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.760010 17226 tablet_service.cc:1511] Processing CreateTablet for tablet 5b2b8f6f05b74c849d19705a2e8ab0e7 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.757748 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.109s user 0.016s sys 0.012s Metrics: {"dirs.queue_time_us":2611,"dirs.run_cpu_time_us":785,"dirs.run_wall_time_us":6375,"lbm_write_time_us":330,"lbm_writes_lt_1ms":7,"num_input_rowsets":0,"thread_start_us":96354,"threads_started":1}
I20260929 02:01:06.760421 16956 tablet_service.cc:1511] Processing CreateTablet for tablet 167d937681dc4e509f1d5d4ce2358d27 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.760892 16956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 167d937681dc4e509f1d5d4ce2358d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.761631 16955 tablet_service.cc:1511] Processing CreateTablet for tablet 5b2b8f6f05b74c849d19705a2e8ab0e7 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.762501 16955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b2b8f6f05b74c849d19705a2e8ab0e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.767191 17092 tablet_service.cc:1511] Processing CreateTablet for tablet 5b2b8f6f05b74c849d19705a2e8ab0e7 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.766660 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.123s user 0.030s sys 0.014s Metrics: {"bytes_written":682,"cfile_cache_hit":16,"cfile_cache_hit_bytes":15247,"cfile_cache_miss":42,"cfile_cache_miss_bytes":24226,"cfile_init":12,"delete_count":0,"delta_iterators_relevant":13,"dirs.queue_time_us":5294,"dirs.run_cpu_time_us":1599,"dirs.run_wall_time_us":10312,"drs_written":1,"lbm_read_time_us":4626,"lbm_reads_1-10_ms":1,"lbm_reads_lt_1ms":89,"lbm_write_time_us":1136,"lbm_writes_lt_1ms":31,"mutex_wait_us":37,"num_input_rowsets":2,"peak_mem_usage":65536,"reinsert_count":0,"rows_written":54,"thread_start_us":21519,"threads_started":2,"update_count":16}
I20260929 02:01:06.767618 17092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b2b8f6f05b74c849d19705a2e8ab0e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.767549 17228 tablet_service.cc:1511] Processing CreateTablet for tablet 925a0eec8efb496ba538b7f7c44d84f2 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.768251 17226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5b2b8f6f05b74c849d19705a2e8ab0e7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.768664 17584 ts_tablet_manager.cc:1962] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.769789 17585 ts_tablet_manager.cc:1975] T 2e9a65b8f4be4f1d99eef1517d20743c P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:06.769935 17585 log.cc:1199] T 2e9a65b8f4be4f1d99eef1517d20743c P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/2e9a65b8f4be4f1d99eef1517d20743c
I20260929 02:01:06.770457 17585 ts_tablet_manager.cc:1996] T 2e9a65b8f4be4f1d99eef1517d20743c P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:06.771625 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 2e9a65b8f4be4f1d99eef1517d20743c (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:06.771937 17228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 925a0eec8efb496ba538b7f7c44d84f2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.775688 17584 ts_tablet_manager.cc:1975] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:06.775852 17584 log.cc:1199] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/a4083e6b1e594ee88efbd7e1f04f70e0
I20260929 02:01:06.776366 17584 ts_tablet_manager.cc:1996] T a4083e6b1e594ee88efbd7e1f04f70e0 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:06.776397 17594 tablet_bootstrap.cc:492] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:06.777783 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet a4083e6b1e594ee88efbd7e1f04f70e0 (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:06.778338 17595 tablet_bootstrap.cc:654] T d9511e2734f1414ba6df2e7c9459ce34 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:06.778434 17584 tablet_replica.cc:333] T d5da16034f2244a2b335f89f6ab7caa9 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:06.778638 17584 raft_consensus.cc:2243] T d5da16034f2244a2b335f89f6ab7caa9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:06.778868 17584 raft_consensus.cc:2272] T d5da16034f2244a2b335f89f6ab7caa9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.778952 17594 tablet_bootstrap.cc:654] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:06.781558 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.149s user 0.011s sys 0.003s Metrics: {"bytes_written":6941,"cfile_init":1,"dirs.queue_time_us":56,"dirs.run_cpu_time_us":150,"dirs.run_wall_time_us":1147,"drs_written":1,"lbm_read_time_us":76,"lbm_reads_lt_1ms":4,"lbm_write_time_us":911,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":17,"spinlock_wait_cycles":248064}
I20260929 02:01:06.782536 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.782868 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260929 02:01:06.783191 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260929 02:01:06.783464 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:06.785768 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":63,"lbm_reads_lt_1ms":4}
I20260929 02:01:06.786208 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.786450 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.788682 17585 tablet_replica.cc:333] T 00f69d45340f4d578a7f3b810c6f9fec P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:06.788928 17585 raft_consensus.cc:2243] T 00f69d45340f4d578a7f3b810c6f9fec P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:06.789113 17585 raft_consensus.cc:2272] T 00f69d45340f4d578a7f3b810c6f9fec P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.789740 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.004s sys 0.000s Metrics: {"bytes_written":682,"delete_count":0,"lbm_write_time_us":225,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":16}
I20260929 02:01:06.790400 17595 tablet_bootstrap.cc:492] T d9511e2734f1414ba6df2e7c9459ce34 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:06.790419 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.342198
I20260929 02:01:06.790563 17595 ts_tablet_manager.cc:1449] T d9511e2734f1414ba6df2e7c9459ce34 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.040s user 0.004s sys 0.000s
I20260929 02:01:06.790740 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.342198
I20260929 02:01:06.790987 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.342198
I20260929 02:01:06.791273 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.342198
I20260929 02:01:06.791438 17595 raft_consensus.cc:359] T d9511e2734f1414ba6df2e7c9459ce34 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.791641 17583 ts_tablet_manager.cc:1975] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260929 02:01:06.791678 17595 raft_consensus.cc:385] T d9511e2734f1414ba6df2e7c9459ce34 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:06.791785 17595 raft_consensus.cc:740] T d9511e2734f1414ba6df2e7c9459ce34 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:06.791838 17583 log.cc:1199] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/8cd30e2805234d9fadc0d2ad839a6bc3
I20260929 02:01:06.792069 17595 consensus_queue.cc:260] T d9511e2734f1414ba6df2e7c9459ce34 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.792223 17583 ts_tablet_manager.cc:1996] T 8cd30e2805234d9fadc0d2ad839a6bc3 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:06.792922 17585 ts_tablet_manager.cc:1962] T 00f69d45340f4d578a7f3b810c6f9fec P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.793524 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 8cd30e2805234d9fadc0d2ad839a6bc3 (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:06.793962 17583 tablet_replica.cc:333] T 3f0398c024bf4cc48ec1670fc847c7ab P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:06.794175 17583 raft_consensus.cc:2243] T 3f0398c024bf4cc48ec1670fc847c7ab P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:06.794391 17583 raft_consensus.cc:2272] T 3f0398c024bf4cc48ec1670fc847c7ab P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.795889 17226 tablet_service.cc:1511] Processing CreateTablet for tablet 167d937681dc4e509f1d5d4ce2358d27 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.796608 17226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 167d937681dc4e509f1d5d4ce2358d27. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.796628 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.796664 17231 tablet_service.cc:1511] Processing CreateTablet for tablet 5acedab1b27c414fa5181e35aa9ac24b (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.796896 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.797067 17231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5acedab1b27c414fa5181e35aa9ac24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.797124 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:06.800022 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=0.316842
I20260929 02:01:06.804718 17595 ts_tablet_manager.cc:1480] T d9511e2734f1414ba6df2e7c9459ce34 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.012s user 0.003s sys 0.000s
I20260929 02:01:06.804782 17233 tablet_service.cc:1511] Processing CreateTablet for tablet c6cac2e61f6f4795869af850549096a9 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.805099 17595 tablet_bootstrap.cc:492] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:06.805060 17225 tablet_service.cc:1511] Processing CreateTablet for tablet ae7f6651c994403697a0f04508b70dae (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.805323 17221 tablet_service.cc:1511] Processing CreateTablet for tablet c7892f83a6e34ba0899bb3ac41a749a5 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.805611 17229 tablet_service.cc:1511] Processing CreateTablet for tablet 306f1984e9324a949db3435c56fcfb8f (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.805853 17223 tablet_service.cc:1511] Processing CreateTablet for tablet bbdc98a938b247bdaf113dc94c3b4d59 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.808183 17222 tablet_service.cc:1511] Processing CreateTablet for tablet 7239b23c6e30483d9bd0738e07c63610 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.808777 17218 tablet_service.cc:1511] Processing CreateTablet for tablet a21bdeb06bb24a45945e00b68b17ca5b (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.809188 17233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6cac2e61f6f4795869af850549096a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.809265 17219 tablet_service.cc:1511] Processing CreateTablet for tablet 2e0146fb337d43b2921fb5fd34fb95ed (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.809788 17232 tablet_service.cc:1511] Processing CreateTablet for tablet b402b4aee2a44330a87657f5a9469dc0 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.810840 17224 tablet_service.cc:1511] Processing CreateTablet for tablet 8f328eef30d541c7a61e7d7403e597c7 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.812171 17595 tablet_bootstrap.cc:654] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:06.812714 17225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae7f6651c994403697a0f04508b70dae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.812645 17220 tablet_service.cc:1511] Processing CreateTablet for tablet 05f61c0da0dd4a20a60cc2b3390cb07c (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.813895 16963 tablet_service.cc:1511] Processing CreateTablet for tablet ae7f6651c994403697a0f04508b70dae (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.814400 17592 tablet_bootstrap.cc:654] T d9511e2734f1414ba6df2e7c9459ce34 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:06.815457 16953 tablet_service.cc:1511] Processing CreateTablet for tablet c6cac2e61f6f4795869af850549096a9 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.816679 17584 ts_tablet_manager.cc:1962] T d5da16034f2244a2b335f89f6ab7caa9 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.820549 16963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae7f6651c994403697a0f04508b70dae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.820783 16961 tablet_service.cc:1511] Processing CreateTablet for tablet 7239b23c6e30483d9bd0738e07c63610 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.820888 16733 tablet.cc:2133] T 00000000000000000000000000000000 P 463404b2470c42c9b6bd67f027cf5e2b: Compaction resulted in no output rows (all input rows were GCed!) Removing all input rowsets.
I20260929 02:01:06.821164 16962 tablet_service.cc:1511] Processing CreateTablet for tablet c7892f83a6e34ba0899bb3ac41a749a5 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.821792 16954 tablet_service.cc:1511] Processing CreateTablet for tablet 306f1984e9324a949db3435c56fcfb8f (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.822839 16964 tablet_service.cc:1511] Processing CreateTablet for tablet 8f328eef30d541c7a61e7d7403e597c7 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.823937 16960 tablet_service.cc:1511] Processing CreateTablet for tablet 2e0146fb337d43b2921fb5fd34fb95ed (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.824827 16965 tablet_service.cc:1511] Processing CreateTablet for tablet a21bdeb06bb24a45945e00b68b17ca5b (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.825440 16961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7239b23c6e30483d9bd0738e07c63610. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.825928 16952 tablet_service.cc:1511] Processing CreateTablet for tablet 05f61c0da0dd4a20a60cc2b3390cb07c (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.825762 16955 tablet_service.cc:1511] Processing CreateTablet for tablet bbdc98a938b247bdaf113dc94c3b4d59 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.826793 16951 tablet_service.cc:1511] Processing CreateTablet for tablet b402b4aee2a44330a87657f5a9469dc0 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.830132 17100 tablet_service.cc:1511] Processing CreateTablet for tablet c7892f83a6e34ba0899bb3ac41a749a5 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.831609 17091 tablet_service.cc:1511] Processing CreateTablet for tablet bbdc98a938b247bdaf113dc94c3b4d59 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.831879 16952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05f61c0da0dd4a20a60cc2b3390cb07c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.836064 17221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7892f83a6e34ba0899bb3ac41a749a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.836561 17090 tablet_service.cc:1511] Processing CreateTablet for tablet 7239b23c6e30483d9bd0738e07c63610 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.837992 16951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b402b4aee2a44330a87657f5a9469dc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.840875 17583 ts_tablet_manager.cc:1962] T 3f0398c024bf4cc48ec1670fc847c7ab P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.841133 16960 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e0146fb337d43b2921fb5fd34fb95ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.841360 16965 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a21bdeb06bb24a45945e00b68b17ca5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.842813 16962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7892f83a6e34ba0899bb3ac41a749a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.844677 16954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 306f1984e9324a949db3435c56fcfb8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.848608 16964 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f328eef30d541c7a61e7d7403e597c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.858165 16956 tablet_service.cc:1511] Processing CreateTablet for tablet 5acedab1b27c414fa5181e35aa9ac24b (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.858706 16956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5acedab1b27c414fa5181e35aa9ac24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.864707 16955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbdc98a938b247bdaf113dc94c3b4d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.868587 17087 tablet_service.cc:1511] Processing CreateTablet for tablet 306f1984e9324a949db3435c56fcfb8f (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.868678 17097 tablet_service.cc:1511] Processing CreateTablet for tablet c6cac2e61f6f4795869af850549096a9 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.869796 17094 tablet_service.cc:1511] Processing CreateTablet for tablet 5acedab1b27c414fa5181e35aa9ac24b (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.870275 17095 tablet_service.cc:1511] Processing CreateTablet for tablet 05f61c0da0dd4a20a60cc2b3390cb07c (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.871470 17085 tablet_service.cc:1511] Processing CreateTablet for tablet b402b4aee2a44330a87657f5a9469dc0 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.872025 17088 tablet_service.cc:1511] Processing CreateTablet for tablet ae7f6651c994403697a0f04508b70dae (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.872475 17098 tablet_service.cc:1511] Processing CreateTablet for tablet 8f328eef30d541c7a61e7d7403e597c7 (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.872998 17096 tablet_service.cc:1511] Processing CreateTablet for tablet a21bdeb06bb24a45945e00b68b17ca5b (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.873098 17092 tablet_service.cc:1511] Processing CreateTablet for tablet 2e0146fb337d43b2921fb5fd34fb95ed (DEFAULT_TABLE table=table-2 [id=73fc6290acff43a9b667eafda6683791]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:06.873423 17100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c7892f83a6e34ba0899bb3ac41a749a5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.873700 17092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e0146fb337d43b2921fb5fd34fb95ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.873942 17095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05f61c0da0dd4a20a60cc2b3390cb07c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.875216 17097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6cac2e61f6f4795869af850549096a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.875689 17091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbdc98a938b247bdaf113dc94c3b4d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.876261 17088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ae7f6651c994403697a0f04508b70dae. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.876621 17096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a21bdeb06bb24a45945e00b68b17ca5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.878065 17085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b402b4aee2a44330a87657f5a9469dc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.879293 17229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 306f1984e9324a949db3435c56fcfb8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.879393 17098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f328eef30d541c7a61e7d7403e597c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.880631 17087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 306f1984e9324a949db3435c56fcfb8f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.882670 17090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7239b23c6e30483d9bd0738e07c63610. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.883744 17094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5acedab1b27c414fa5181e35aa9ac24b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.887153 17584 ts_tablet_manager.cc:1975] T d5da16034f2244a2b335f89f6ab7caa9 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:06.889889 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.089s user 0.009s sys 0.000s Metrics: {"dirs.queue_time_us":2515,"dirs.run_cpu_time_us":936,"dirs.run_wall_time_us":10934,"lbm_write_time_us":428,"lbm_writes_lt_1ms":7,"mutex_wait_us":63951,"num_input_rowsets":0}
W20260929 02:01:06.890480 16921 consensus_peers.cc:597] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f -> Peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Couldn't send request to peer d103dd4823044faebe46100a0dfbee88. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e9a65b8f4be4f1d99eef1517d20743c. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:01:06.890674 16953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c6cac2e61f6f4795869af850549096a9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.892798 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.101s user 0.019s sys 0.000s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":1662,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":1306,"lbm_read_time_us":516,"lbm_reads_lt_1ms":21,"lbm_write_time_us":395,"lbm_writes_lt_1ms":9,"peak_mem_usage":32768,"reinsert_count":0,"update_count":16}
I20260929 02:01:06.893898 17584 log.cc:1199] T d5da16034f2244a2b335f89f6ab7caa9 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/d5da16034f2244a2b335f89f6ab7caa9
I20260929 02:01:06.896847 17220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 05f61c0da0dd4a20a60cc2b3390cb07c. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.898330 17584 ts_tablet_manager.cc:1996] T d5da16034f2244a2b335f89f6ab7caa9 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:06.901074 17219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e0146fb337d43b2921fb5fd34fb95ed. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.904748 17218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a21bdeb06bb24a45945e00b68b17ca5b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.906243 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet d5da16034f2244a2b335f89f6ab7caa9 (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:06.906910 17223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet bbdc98a938b247bdaf113dc94c3b4d59. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.909369 17584 tablet_replica.cc:333] T 85b63af0846742c99534e9bd449181d3 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:06.909611 17584 raft_consensus.cc:2243] T 85b63af0846742c99534e9bd449181d3 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:06.909790 17584 raft_consensus.cc:2272] T 85b63af0846742c99534e9bd449181d3 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.910543 17222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 7239b23c6e30483d9bd0738e07c63610. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.916823 17224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f328eef30d541c7a61e7d7403e597c7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.917526 17232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b402b4aee2a44330a87657f5a9469dc0. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:06.930330 17595 tablet_bootstrap.cc:492] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:06.936532 17584 ts_tablet_manager.cc:1962] T 85b63af0846742c99534e9bd449181d3 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.938184 17595 ts_tablet_manager.cc:1449] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.125s user 0.006s sys 0.009s
I20260929 02:01:06.953648 17595 raft_consensus.cc:359] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.955044 17594 tablet_bootstrap.cc:492] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:06.955276 17594 ts_tablet_manager.cc:1449] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.179s user 0.004s sys 0.013s
I20260929 02:01:06.965005 17583 ts_tablet_manager.cc:1975] T 3f0398c024bf4cc48ec1670fc847c7ab P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:06.965160 17583 log.cc:1199] T 3f0398c024bf4cc48ec1670fc847c7ab P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/3f0398c024bf4cc48ec1670fc847c7ab
I20260929 02:01:06.966142 17583 ts_tablet_manager.cc:1996] T 3f0398c024bf4cc48ec1670fc847c7ab P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:06.967407 17583 tablet_replica.cc:333] T 00f69d45340f4d578a7f3b810c6f9fec P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:06.967610 17583 raft_consensus.cc:2243] T 00f69d45340f4d578a7f3b810c6f9fec P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:06.967792 17583 raft_consensus.cc:2272] T 00f69d45340f4d578a7f3b810c6f9fec P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.967787 17595 raft_consensus.cc:385] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:06.968230 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 3f0398c024bf4cc48ec1670fc847c7ab (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:06.969220 17583 ts_tablet_manager.cc:1962] T 00f69d45340f4d578a7f3b810c6f9fec P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.967967 17595 raft_consensus.cc:740] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:06.973174 17592 tablet_bootstrap.cc:492] T d9511e2734f1414ba6df2e7c9459ce34 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:06.973431 17592 ts_tablet_manager.cc:1449] T d9511e2734f1414ba6df2e7c9459ce34 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.216s user 0.027s sys 0.011s
I20260929 02:01:06.973448 17594 raft_consensus.cc:359] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.973683 17594 raft_consensus.cc:385] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:06.974426 17594 raft_consensus.cc:740] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:06.974709 17594 consensus_queue.cc:260] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.974800 17592 raft_consensus.cc:359] T d9511e2734f1414ba6df2e7c9459ce34 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.976326 17592 raft_consensus.cc:385] T d9511e2734f1414ba6df2e7c9459ce34 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:06.976984 17592 raft_consensus.cc:740] T d9511e2734f1414ba6df2e7c9459ce34 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:06.978266 17592 consensus_queue.cc:260] T d9511e2734f1414ba6df2e7c9459ce34 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.980592 17592 ts_tablet_manager.cc:1480] T d9511e2734f1414ba6df2e7c9459ce34 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.007s user 0.001s sys 0.001s
I20260929 02:01:06.981652 17583 ts_tablet_manager.cc:1975] T 00f69d45340f4d578a7f3b810c6f9fec P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:06.981846 17583 log.cc:1199] T 00f69d45340f4d578a7f3b810c6f9fec P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/00f69d45340f4d578a7f3b810c6f9fec
I20260929 02:01:06.982352 17583 ts_tablet_manager.cc:1996] T 00f69d45340f4d578a7f3b810c6f9fec P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:06.972757 17595 consensus_queue.cc:260] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:06.983531 17595 ts_tablet_manager.cc:1480] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.045s user 0.003s sys 0.000s
I20260929 02:01:06.983587 17594 ts_tablet_manager.cc:1480] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.028s user 0.000s sys 0.003s
I20260929 02:01:06.983916 17592 tablet_bootstrap.cc:492] T 925a0eec8efb496ba538b7f7c44d84f2 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:06.986018 17592 tablet_bootstrap.cc:654] T 925a0eec8efb496ba538b7f7c44d84f2 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:06.986644 17595 tablet_bootstrap.cc:492] T 167d937681dc4e509f1d5d4ce2358d27 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:06.988375 17585 ts_tablet_manager.cc:1975] T 00f69d45340f4d578a7f3b810c6f9fec P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:06.988572 17585 log.cc:1199] T 00f69d45340f4d578a7f3b810c6f9fec P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/00f69d45340f4d578a7f3b810c6f9fec
I20260929 02:01:06.989053 17585 ts_tablet_manager.cc:1996] T 00f69d45340f4d578a7f3b810c6f9fec P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:06.990149 17585 tablet_replica.cc:333] T a4083e6b1e594ee88efbd7e1f04f70e0 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:06.990391 17585 raft_consensus.cc:2243] T a4083e6b1e594ee88efbd7e1f04f70e0 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:06.990592 17585 raft_consensus.cc:2272] T a4083e6b1e594ee88efbd7e1f04f70e0 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.992259 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 00f69d45340f4d578a7f3b810c6f9fec (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:06.992754 17585 ts_tablet_manager.cc:1962] T a4083e6b1e594ee88efbd7e1f04f70e0 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.993849 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 00f69d45340f4d578a7f3b810c6f9fec (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:06.994269 17583 tablet_replica.cc:333] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:06.994607 17583 raft_consensus.cc:2243] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:06.995079 17583 raft_consensus.cc:2272] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:06.995260 17594 tablet_bootstrap.cc:492] T 167d937681dc4e509f1d5d4ce2358d27 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:06.996280 17583 ts_tablet_manager.cc:1962] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:06.998485 17594 tablet_bootstrap.cc:654] T 167d937681dc4e509f1d5d4ce2358d27 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.000735 17585 ts_tablet_manager.cc:1975] T a4083e6b1e594ee88efbd7e1f04f70e0 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:07.000895 17585 log.cc:1199] T a4083e6b1e594ee88efbd7e1f04f70e0 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/a4083e6b1e594ee88efbd7e1f04f70e0
I20260929 02:01:07.001367 17585 ts_tablet_manager.cc:1996] T a4083e6b1e594ee88efbd7e1f04f70e0 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:07.003510 17595 tablet_bootstrap.cc:654] T 167d937681dc4e509f1d5d4ce2358d27 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.004268 17583 ts_tablet_manager.cc:1975] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:07.004505 17583 log.cc:1199] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/2e9a65b8f4be4f1d99eef1517d20743c
I20260929 02:01:07.005054 17583 ts_tablet_manager.cc:1996] T 2e9a65b8f4be4f1d99eef1517d20743c P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:07.006412 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 2e9a65b8f4be4f1d99eef1517d20743c (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:07.006868 17583 tablet_replica.cc:333] T a4083e6b1e594ee88efbd7e1f04f70e0 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:07.007166 17583 raft_consensus.cc:2243] T a4083e6b1e594ee88efbd7e1f04f70e0 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:07.007452 17583 raft_consensus.cc:2272] T a4083e6b1e594ee88efbd7e1f04f70e0 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:07.009389 17583 ts_tablet_manager.cc:1962] T a4083e6b1e594ee88efbd7e1f04f70e0 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:07.010016 17543 raft_consensus.cc:493] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:07.010257 17543 raft_consensus.cc:515] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.011211 17543 leader_election.cc:290] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:07.014663 17585 tablet_replica.cc:333] T d5da16034f2244a2b335f89f6ab7caa9 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:07.014823 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet a4083e6b1e594ee88efbd7e1f04f70e0 (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:07.014914 17585 raft_consensus.cc:2243] T d5da16034f2244a2b335f89f6ab7caa9 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:07.015223 17585 raft_consensus.cc:2272] T d5da16034f2244a2b335f89f6ab7caa9 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:07.015666 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9511e2734f1414ba6df2e7c9459ce34" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:07.015869 16986 raft_consensus.cc:2468] T d9511e2734f1414ba6df2e7c9459ce34 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:07.016623 17056 leader_election.cc:304] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:07.017079 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9511e2734f1414ba6df2e7c9459ce34" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:07.017284 17585 ts_tablet_manager.cc:1962] T d5da16034f2244a2b335f89f6ab7caa9 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:07.017449 17398 raft_consensus.cc:2804] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:07.017293 17253 raft_consensus.cc:2468] T d9511e2734f1414ba6df2e7c9459ce34 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:07.017575 17398 raft_consensus.cc:493] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:07.017684 17398 raft_consensus.cc:3060] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.019426 17595 tablet_bootstrap.cc:492] T 167d937681dc4e509f1d5d4ce2358d27 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:07.019575 17595 ts_tablet_manager.cc:1449] T 167d937681dc4e509f1d5d4ce2358d27 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.033s user 0.003s sys 0.000s
I20260929 02:01:07.020385 17595 raft_consensus.cc:359] T 167d937681dc4e509f1d5d4ce2358d27 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.020581 17595 raft_consensus.cc:385] T 167d937681dc4e509f1d5d4ce2358d27 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.020711 17595 raft_consensus.cc:740] T 167d937681dc4e509f1d5d4ce2358d27 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.020896 17398 raft_consensus.cc:515] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.021057 17595 consensus_queue.cc:260] T 167d937681dc4e509f1d5d4ce2358d27 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.021642 17595 ts_tablet_manager.cc:1480] T 167d937681dc4e509f1d5d4ce2358d27 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 02:01:07.022274 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9511e2734f1414ba6df2e7c9459ce34" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:07.022462 16986 raft_consensus.cc:3060] T d9511e2734f1414ba6df2e7c9459ce34 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.021958 17595 tablet_bootstrap.cc:492] T 925a0eec8efb496ba538b7f7c44d84f2 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:07.026232 16986 raft_consensus.cc:2468] T d9511e2734f1414ba6df2e7c9459ce34 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:07.027136 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d9511e2734f1414ba6df2e7c9459ce34" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:07.027339 17253 raft_consensus.cc:3060] T d9511e2734f1414ba6df2e7c9459ce34 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.030335 17253 raft_consensus.cc:2468] T d9511e2734f1414ba6df2e7c9459ce34 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:07.031306 17055 leader_election.cc:304] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:07.031766 17543 raft_consensus.cc:2804] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:07.031948 17543 raft_consensus.cc:697] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Becoming Leader. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Running, Role: LEADER
I20260929 02:01:07.031994 17585 ts_tablet_manager.cc:1975] T d5da16034f2244a2b335f89f6ab7caa9 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:07.032131 17585 log.cc:1199] T d5da16034f2244a2b335f89f6ab7caa9 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/d5da16034f2244a2b335f89f6ab7caa9
I20260929 02:01:07.032230 17543 consensus_queue.cc:237] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.032617 17585 ts_tablet_manager.cc:1996] T d5da16034f2244a2b335f89f6ab7caa9 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:07.032714 17595 tablet_bootstrap.cc:654] T 925a0eec8efb496ba538b7f7c44d84f2 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.032804 17584 ts_tablet_manager.cc:1975] T 85b63af0846742c99534e9bd449181d3 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:07.032969 17584 log.cc:1199] T 85b63af0846742c99534e9bd449181d3 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/85b63af0846742c99534e9bd449181d3
I20260929 02:01:07.033427 17584 ts_tablet_manager.cc:1996] T 85b63af0846742c99534e9bd449181d3 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:07.033910 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet d5da16034f2244a2b335f89f6ab7caa9 (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:07.034453 17585 tablet_replica.cc:333] T 85b63af0846742c99534e9bd449181d3 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:07.034715 17585 raft_consensus.cc:2243] T 85b63af0846742c99534e9bd449181d3 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:07.034706 17398 leader_election.cc:290] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:07.034932 17585 raft_consensus.cc:2272] T 85b63af0846742c99534e9bd449181d3 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:07.035806 17585 ts_tablet_manager.cc:1962] T 85b63af0846742c99534e9bd449181d3 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:07.037017 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 85b63af0846742c99534e9bd449181d3 (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:07.037573 17584 tablet_replica.cc:333] T 5e7d7ccf81494fec9e066824c69abd61 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:07.030678 17583 ts_tablet_manager.cc:1975] T a4083e6b1e594ee88efbd7e1f04f70e0 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:07.031853 17592 tablet_bootstrap.cc:492] T 925a0eec8efb496ba538b7f7c44d84f2 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:07.037890 17592 ts_tablet_manager.cc:1449] T 925a0eec8efb496ba538b7f7c44d84f2 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.054s user 0.003s sys 0.015s
I20260929 02:01:07.038180 17584 raft_consensus.cc:2243] T 5e7d7ccf81494fec9e066824c69abd61 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:07.038681 17584 raft_consensus.cc:2272] T 5e7d7ccf81494fec9e066824c69abd61 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:07.039012 17592 raft_consensus.cc:359] T 925a0eec8efb496ba538b7f7c44d84f2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.039227 17592 raft_consensus.cc:385] T 925a0eec8efb496ba538b7f7c44d84f2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.039335 17592 raft_consensus.cc:740] T 925a0eec8efb496ba538b7f7c44d84f2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.040091 17584 ts_tablet_manager.cc:1962] T 5e7d7ccf81494fec9e066824c69abd61 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:07.040093 17592 consensus_queue.cc:260] T 925a0eec8efb496ba538b7f7c44d84f2 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.040777 17592 ts_tablet_manager.cc:1480] T 925a0eec8efb496ba538b7f7c44d84f2 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260929 02:01:07.041453 17583 log.cc:1199] T a4083e6b1e594ee88efbd7e1f04f70e0 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/a4083e6b1e594ee88efbd7e1f04f70e0
I20260929 02:01:07.042148 17583 ts_tablet_manager.cc:1996] T a4083e6b1e594ee88efbd7e1f04f70e0 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:07.043103 17583 tablet_replica.cc:333] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:07.043152 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet a4083e6b1e594ee88efbd7e1f04f70e0 (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:07.043360 17583 raft_consensus.cc:2243] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:07.043720 17583 raft_consensus.cc:2272] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:07.044728 17592 tablet_bootstrap.cc:492] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:07.048044 17585 ts_tablet_manager.cc:1975] T 85b63af0846742c99534e9bd449181d3 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:07.048195 17585 log.cc:1199] T 85b63af0846742c99534e9bd449181d3 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/85b63af0846742c99534e9bd449181d3
I20260929 02:01:07.049199 17583 ts_tablet_manager.cc:1962] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:07.049935 17592 tablet_bootstrap.cc:654] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.052047 16818 catalog_manager.cc:5838] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66). New cstate: current_term: 1 leader_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:07.056274 17585 ts_tablet_manager.cc:1996] T 85b63af0846742c99534e9bd449181d3 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:07.059194 17583 ts_tablet_manager.cc:1975] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:07.059412 17583 log.cc:1199] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/d5da16034f2244a2b335f89f6ab7caa9
I20260929 02:01:07.059958 17583 ts_tablet_manager.cc:1996] T d5da16034f2244a2b335f89f6ab7caa9 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:07.060971 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 85b63af0846742c99534e9bd449181d3 (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:07.061678 17585 tablet_replica.cc:333] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:07.061925 17585 raft_consensus.cc:2243] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:07.062078 17583 tablet_replica.cc:333] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:07.062294 17585 raft_consensus.cc:2272] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:07.062675 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet d5da16034f2244a2b335f89f6ab7caa9 (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:07.062927 17583 raft_consensus.cc:2243] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:07.063373 17583 raft_consensus.cc:2272] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:07.064465 17583 ts_tablet_manager.cc:1962] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:07.066895 17584 ts_tablet_manager.cc:1975] T 5e7d7ccf81494fec9e066824c69abd61 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:07.067050 17584 log.cc:1199] T 5e7d7ccf81494fec9e066824c69abd61 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/5e7d7ccf81494fec9e066824c69abd61
I20260929 02:01:07.067639 17584 ts_tablet_manager.cc:1996] T 5e7d7ccf81494fec9e066824c69abd61 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:07.068176 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.068262 17595 tablet_bootstrap.cc:492] T 925a0eec8efb496ba538b7f7c44d84f2 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:07.068560 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.068817 17595 ts_tablet_manager.cc:1449] T 925a0eec8efb496ba538b7f7c44d84f2 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.047s user 0.003s sys 0.000s
I20260929 02:01:07.069211 17585 ts_tablet_manager.cc:1962] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:07.070065 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.070767 17594 tablet_bootstrap.cc:492] T 167d937681dc4e509f1d5d4ce2358d27 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:07.071024 17594 ts_tablet_manager.cc:1449] T 167d937681dc4e509f1d5d4ce2358d27 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.083s user 0.001s sys 0.009s
I20260929 02:01:07.069752 17595 raft_consensus.cc:359] T 925a0eec8efb496ba538b7f7c44d84f2 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.071363 17595 raft_consensus.cc:385] T 925a0eec8efb496ba538b7f7c44d84f2 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.072154 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.072295 17594 raft_consensus.cc:359] T 167d937681dc4e509f1d5d4ce2358d27 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.072460 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.072543 17594 raft_consensus.cc:385] T 167d937681dc4e509f1d5d4ce2358d27 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.072628 17594 raft_consensus.cc:740] T 167d937681dc4e509f1d5d4ce2358d27 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.072858 17595 raft_consensus.cc:740] T 925a0eec8efb496ba538b7f7c44d84f2 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.072870 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.073239 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.073204 17595 consensus_queue.cc:260] T 925a0eec8efb496ba538b7f7c44d84f2 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.073520 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:07.073772 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:07.074141 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.074815 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.077642 17583 ts_tablet_manager.cc:1975] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:07.077893 17583 log.cc:1199] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/85b63af0846742c99534e9bd449181d3
I20260929 02:01:07.078510 17583 ts_tablet_manager.cc:1996] T 85b63af0846742c99534e9bd449181d3 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:07.078883 17595 ts_tablet_manager.cc:1480] T 925a0eec8efb496ba538b7f7c44d84f2 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.010s user 0.001s sys 0.001s
I20260929 02:01:07.080587 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 85b63af0846742c99534e9bd449181d3 (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:07.081060 17583 tablet_replica.cc:333] T 5e7d7ccf81494fec9e066824c69abd61 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:07.081327 17583 raft_consensus.cc:2243] T 5e7d7ccf81494fec9e066824c69abd61 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:07.081601 17583 raft_consensus.cc:2272] T 5e7d7ccf81494fec9e066824c69abd61 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:07.082033 17592 tablet_bootstrap.cc:492] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:07.082216 17592 ts_tablet_manager.cc:1449] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.038s user 0.006s sys 0.007s
I20260929 02:01:07.082736 17583 ts_tablet_manager.cc:1962] T 5e7d7ccf81494fec9e066824c69abd61 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:07.083128 17592 raft_consensus.cc:359] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.083267 17594 consensus_queue.cc:260] T 167d937681dc4e509f1d5d4ce2358d27 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.083370 17592 raft_consensus.cc:385] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.083549 17592 raft_consensus.cc:740] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.083900 17595 tablet_bootstrap.cc:492] T c6cac2e61f6f4795869af850549096a9 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:07.083797 17592 consensus_queue.cc:260] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.085892 17592 ts_tablet_manager.cc:1480] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260929 02:01:07.086210 17592 tablet_bootstrap.cc:492] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:07.086952 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.014s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":183,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:07.087538 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.019s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":157,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:07.088477 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.014s user 0.004s sys 0.004s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":916,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":1449,"dirs.run_cpu_time_us":138,"dirs.run_wall_time_us":1253,"lbm_read_time_us":194,"lbm_reads_lt_1ms":8,"lbm_write_time_us":199,"lbm_writes_lt_1ms":4,"mutex_wait_us":4745,"reinsert_count":0,"update_count":17}
I20260929 02:01:07.090507 17592 tablet_bootstrap.cc:654] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.091719 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 5e7d7ccf81494fec9e066824c69abd61 (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:07.092139 17584 tablet_replica.cc:333] T 044313d0ac094292808355c152d310aa P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:07.092489 17584 raft_consensus.cc:2243] T 044313d0ac094292808355c152d310aa P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:07.092754 17584 raft_consensus.cc:2272] T 044313d0ac094292808355c152d310aa P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:07.094115 17584 ts_tablet_manager.cc:1962] T 044313d0ac094292808355c152d310aa P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:07.095703 17595 tablet_bootstrap.cc:654] T c6cac2e61f6f4795869af850549096a9 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.096187 17585 ts_tablet_manager.cc:1975] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:07.096419 17585 log.cc:1199] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/5e7d7ccf81494fec9e066824c69abd61
I20260929 02:01:07.094153 17594 ts_tablet_manager.cc:1480] T 167d937681dc4e509f1d5d4ce2358d27 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.023s user 0.002s sys 0.000s
I20260929 02:01:07.096935 17585 ts_tablet_manager.cc:1996] T 5e7d7ccf81494fec9e066824c69abd61 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:07.098958 17583 ts_tablet_manager.cc:1975] T 5e7d7ccf81494fec9e066824c69abd61 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:07.099766 17585 tablet_replica.cc:333] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:07.099906 17594 tablet_bootstrap.cc:492] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:07.099880 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 5e7d7ccf81494fec9e066824c69abd61 (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:07.099992 17585 raft_consensus.cc:2243] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:07.100373 17585 raft_consensus.cc:2272] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:07.101854 17585 ts_tablet_manager.cc:1962] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:07.105247 17583 log.cc:1199] T 5e7d7ccf81494fec9e066824c69abd61 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/5e7d7ccf81494fec9e066824c69abd61
I20260929 02:01:07.105851 17594 tablet_bootstrap.cc:654] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.106444 17583 ts_tablet_manager.cc:1996] T 5e7d7ccf81494fec9e066824c69abd61 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:07.107659 17592 tablet_bootstrap.cc:492] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:07.107904 17592 ts_tablet_manager.cc:1449] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.022s user 0.006s sys 0.001s
I20260929 02:01:07.108917 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 5e7d7ccf81494fec9e066824c69abd61 (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:07.109036 17583 tablet_replica.cc:333] T 044313d0ac094292808355c152d310aa P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:07.109248 17583 raft_consensus.cc:2243] T 044313d0ac094292808355c152d310aa P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:07.109290 17592 raft_consensus.cc:359] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.110973 17583 raft_consensus.cc:2272] T 044313d0ac094292808355c152d310aa P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:07.111064 17592 raft_consensus.cc:385] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.111409 17592 raft_consensus.cc:740] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.111845 17592 consensus_queue.cc:260] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.112434 17583 ts_tablet_manager.cc:1962] T 044313d0ac094292808355c152d310aa P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:07.114202 17585 ts_tablet_manager.cc:1975] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:07.114365 17585 log.cc:1199] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/044313d0ac094292808355c152d310aa
I20260929 02:01:07.114898 17585 ts_tablet_manager.cc:1996] T 044313d0ac094292808355c152d310aa P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:07.116163 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 044313d0ac094292808355c152d310aa (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:07.116652 17585 tablet_replica.cc:333] T 7f3e8d6962a940868e1ab2caa707cff1 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:07.117049 17585 raft_consensus.cc:2243] T 7f3e8d6962a940868e1ab2caa707cff1 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:07.117614 17592 ts_tablet_manager.cc:1480] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.010s user 0.000s sys 0.002s
I20260929 02:01:07.120085 17592 tablet_bootstrap.cc:492] T 2e0146fb337d43b2921fb5fd34fb95ed P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:07.120162 17584 ts_tablet_manager.cc:1975] T 044313d0ac094292808355c152d310aa P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:07.120409 17585 raft_consensus.cc:2272] T 7f3e8d6962a940868e1ab2caa707cff1 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:07.120420 17584 log.cc:1199] T 044313d0ac094292808355c152d310aa P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/044313d0ac094292808355c152d310aa
I20260929 02:01:07.121034 17584 ts_tablet_manager.cc:1996] T 044313d0ac094292808355c152d310aa P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:07.121641 17585 ts_tablet_manager.cc:1962] T 7f3e8d6962a940868e1ab2caa707cff1 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:07.122195 17592 tablet_bootstrap.cc:654] T 2e0146fb337d43b2921fb5fd34fb95ed P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.124392 17594 tablet_bootstrap.cc:492] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:07.124665 17594 ts_tablet_manager.cc:1449] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.028s user 0.004s sys 0.001s
I20260929 02:01:07.125756 17592 tablet_bootstrap.cc:492] T 2e0146fb337d43b2921fb5fd34fb95ed P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:07.125671 17594 raft_consensus.cc:359] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.126008 17594 raft_consensus.cc:385] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.126165 17594 raft_consensus.cc:740] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.126585 17594 consensus_queue.cc:260] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.127382 17594 ts_tablet_manager.cc:1480] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:01:07.128605 17594 tablet_bootstrap.cc:492] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:07.130836 17594 tablet_bootstrap.cc:654] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.131639 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 044313d0ac094292808355c152d310aa (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:07.131511 17584 tablet_replica.cc:333] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:07.132881 17584 raft_consensus.cc:2243] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:07.132930 17592 ts_tablet_manager.cc:1449] T 2e0146fb337d43b2921fb5fd34fb95ed P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.013s user 0.003s sys 0.001s
I20260929 02:01:07.133335 17584 raft_consensus.cc:2272] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:07.133802 17595 tablet_bootstrap.cc:492] T c6cac2e61f6f4795869af850549096a9 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:07.133996 17595 ts_tablet_manager.cc:1449] T c6cac2e61f6f4795869af850549096a9 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.050s user 0.004s sys 0.000s
I20260929 02:01:07.134756 17584 ts_tablet_manager.cc:1962] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:07.134964 17595 raft_consensus.cc:359] T c6cac2e61f6f4795869af850549096a9 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.135214 17595 raft_consensus.cc:385] T c6cac2e61f6f4795869af850549096a9 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.135351 17595 raft_consensus.cc:740] T c6cac2e61f6f4795869af850549096a9 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.135704 17595 consensus_queue.cc:260] T c6cac2e61f6f4795869af850549096a9 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.136317 17595 ts_tablet_manager.cc:1480] T c6cac2e61f6f4795869af850549096a9 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:01:07.136677 17595 tablet_bootstrap.cc:492] T c7892f83a6e34ba0899bb3ac41a749a5 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:07.139765 17585 ts_tablet_manager.cc:1975] T 7f3e8d6962a940868e1ab2caa707cff1 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:07.139964 17585 log.cc:1199] T 7f3e8d6962a940868e1ab2caa707cff1 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/7f3e8d6962a940868e1ab2caa707cff1
I20260929 02:01:07.140435 17585 ts_tablet_manager.cc:1996] T 7f3e8d6962a940868e1ab2caa707cff1 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:07.141261 17583 ts_tablet_manager.cc:1975] T 044313d0ac094292808355c152d310aa P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:07.141683 17585 tablet_replica.cc:333] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:07.141799 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 7f3e8d6962a940868e1ab2caa707cff1 (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:07.141909 17585 raft_consensus.cc:2243] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:07.142295 17585 raft_consensus.cc:2272] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:07.142803 17592 raft_consensus.cc:359] T 2e0146fb337d43b2921fb5fd34fb95ed P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.143402 17592 raft_consensus.cc:385] T 2e0146fb337d43b2921fb5fd34fb95ed P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.143555 17592 raft_consensus.cc:740] T 2e0146fb337d43b2921fb5fd34fb95ed P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.143944 17592 consensus_queue.cc:260] T 2e0146fb337d43b2921fb5fd34fb95ed P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.144133 17585 ts_tablet_manager.cc:1962] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:07.144155 17594 tablet_bootstrap.cc:492] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:07.144344 17594 ts_tablet_manager.cc:1449] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.016s user 0.003s sys 0.000s
I20260929 02:01:07.145068 17595 tablet_bootstrap.cc:654] T c7892f83a6e34ba0899bb3ac41a749a5 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.145210 17594 raft_consensus.cc:359] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.145504 17594 raft_consensus.cc:385] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.145637 17594 raft_consensus.cc:740] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.145994 17594 consensus_queue.cc:260] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.146235 17592 ts_tablet_manager.cc:1480] T 2e0146fb337d43b2921fb5fd34fb95ed P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260929 02:01:07.146569 17592 tablet_bootstrap.cc:492] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:07.146711 17594 ts_tablet_manager.cc:1480] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:01:07.147291 17594 tablet_bootstrap.cc:492] T 5acedab1b27c414fa5181e35aa9ac24b P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:07.149822 17592 tablet_bootstrap.cc:654] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.150911 17583 log.cc:1199] T 044313d0ac094292808355c152d310aa P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/044313d0ac094292808355c152d310aa
I20260929 02:01:07.151432 17583 ts_tablet_manager.cc:1996] T 044313d0ac094292808355c152d310aa P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:07.151400 17382 raft_consensus.cc:493] T c6cac2e61f6f4795869af850549096a9 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:07.151595 17382 raft_consensus.cc:515] T c6cac2e61f6f4795869af850549096a9 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.152338 17382 leader_election.cc:290] T c6cac2e61f6f4795869af850549096a9 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:07.152493 17583 tablet_replica.cc:333] T 7f3e8d6962a940868e1ab2caa707cff1 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:07.152675 17594 tablet_bootstrap.cc:654] T 5acedab1b27c414fa5181e35aa9ac24b P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.152695 17583 raft_consensus.cc:2243] T 7f3e8d6962a940868e1ab2caa707cff1 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:07.152901 17583 raft_consensus.cc:2272] T 7f3e8d6962a940868e1ab2caa707cff1 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:07.153196 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6cac2e61f6f4795869af850549096a9" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
W20260929 02:01:07.153854 17186 leader_election.cc:343] T c6cac2e61f6f4795869af850549096a9 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.155133 17585 ts_tablet_manager.cc:1975] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:07.155283 17585 log.cc:1199] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/a3f4dfb792b946728e051ae1791cb75a
I20260929 02:01:07.156055 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 044313d0ac094292808355c152d310aa (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:07.156281 17585 ts_tablet_manager.cc:1996] T a3f4dfb792b946728e051ae1791cb75a P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:07.158633 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet a3f4dfb792b946728e051ae1791cb75a (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:07.160663 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6cac2e61f6f4795869af850549096a9" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
W20260929 02:01:07.161350 17189 leader_election.cc:343] T c6cac2e61f6f4795869af850549096a9 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.161523 17189 leader_election.cc:304] T c6cac2e61f6f4795869af850549096a9 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f
I20260929 02:01:07.162046 17382 raft_consensus.cc:2749] T c6cac2e61f6f4795869af850549096a9 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:07.167690 17595 tablet_bootstrap.cc:492] T c7892f83a6e34ba0899bb3ac41a749a5 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:07.167891 17595 ts_tablet_manager.cc:1449] T c7892f83a6e34ba0899bb3ac41a749a5 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.031s user 0.003s sys 0.004s
I20260929 02:01:07.168753 17583 ts_tablet_manager.cc:1962] T 7f3e8d6962a940868e1ab2caa707cff1 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:07.169049 17595 raft_consensus.cc:359] T c7892f83a6e34ba0899bb3ac41a749a5 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.169339 17595 raft_consensus.cc:385] T c7892f83a6e34ba0899bb3ac41a749a5 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.169440 17595 raft_consensus.cc:740] T c7892f83a6e34ba0899bb3ac41a749a5 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.169823 17595 consensus_queue.cc:260] T c7892f83a6e34ba0899bb3ac41a749a5 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.179836 17584 ts_tablet_manager.cc:1975] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:07.180670 17595 ts_tablet_manager.cc:1480] T c7892f83a6e34ba0899bb3ac41a749a5 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.013s user 0.003s sys 0.000s
I20260929 02:01:07.184857 17543 raft_consensus.cc:493] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:07.186003 17543 raft_consensus.cc:515] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.186982 17543 leader_election.cc:290] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:07.187477 17595 tablet_bootstrap.cc:492] T ae7f6651c994403697a0f04508b70dae P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:07.190274 17595 tablet_bootstrap.cc:654] T ae7f6651c994403697a0f04508b70dae P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.192744 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "925a0eec8efb496ba538b7f7c44d84f2" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:07.193449 17594 tablet_bootstrap.cc:492] T 5acedab1b27c414fa5181e35aa9ac24b P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:07.193439 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "925a0eec8efb496ba538b7f7c44d84f2" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:07.193681 17253 raft_consensus.cc:2468] T 925a0eec8efb496ba538b7f7c44d84f2 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:07.193737 17594 ts_tablet_manager.cc:1449] T 5acedab1b27c414fa5181e35aa9ac24b P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.047s user 0.005s sys 0.027s
I20260929 02:01:07.194041 16986 raft_consensus.cc:2468] T 925a0eec8efb496ba538b7f7c44d84f2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:07.194612 17056 leader_election.cc:304] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:07.194901 17594 raft_consensus.cc:359] T 5acedab1b27c414fa5181e35aa9ac24b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.195184 17594 raft_consensus.cc:385] T 5acedab1b27c414fa5181e35aa9ac24b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.195371 17543 raft_consensus.cc:2804] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:07.195478 17543 raft_consensus.cc:493] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:07.195590 17543 raft_consensus.cc:3060] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.198717 17543 raft_consensus.cc:515] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.199939 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "925a0eec8efb496ba538b7f7c44d84f2" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:07.200165 16986 raft_consensus.cc:3060] T 925a0eec8efb496ba538b7f7c44d84f2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.203234 17592 tablet_bootstrap.cc:492] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:07.203819 17592 ts_tablet_manager.cc:1449] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.057s user 0.000s sys 0.008s
I20260929 02:01:07.204516 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "925a0eec8efb496ba538b7f7c44d84f2" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:07.204743 17253 raft_consensus.cc:3060] T 925a0eec8efb496ba538b7f7c44d84f2 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.204720 17592 raft_consensus.cc:359] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.204985 17584 log.cc:1199] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/7f3e8d6962a940868e1ab2caa707cff1
I20260929 02:01:07.205451 17584 ts_tablet_manager.cc:1996] T 7f3e8d6962a940868e1ab2caa707cff1 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:07.206601 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 7f3e8d6962a940868e1ab2caa707cff1 (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:07.207937 17253 raft_consensus.cc:2468] T 925a0eec8efb496ba538b7f7c44d84f2 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:07.208913 17055 leader_election.cc:304] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:07.209192 17584 tablet_replica.cc:333] T a3f4dfb792b946728e051ae1791cb75a P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:07.209219 17398 raft_consensus.cc:2804] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:07.209352 17398 raft_consensus.cc:697] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Becoming Leader. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Running, Role: LEADER
I20260929 02:01:07.209417 17584 raft_consensus.cc:2243] T a3f4dfb792b946728e051ae1791cb75a P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:07.209573 17584 raft_consensus.cc:2272] T a3f4dfb792b946728e051ae1791cb75a P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:07.209673 17398 consensus_queue.cc:237] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.211439 17592 raft_consensus.cc:385] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.212527 16986 raft_consensus.cc:2468] T 925a0eec8efb496ba538b7f7c44d84f2 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:07.212838 17592 raft_consensus.cc:740] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.213853 17543 leader_election.cc:290] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:07.195363 17594 raft_consensus.cc:740] T 5acedab1b27c414fa5181e35aa9ac24b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.218417 17594 consensus_queue.cc:260] T 5acedab1b27c414fa5181e35aa9ac24b P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.219484 17594 ts_tablet_manager.cc:1480] T 5acedab1b27c414fa5181e35aa9ac24b P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.026s user 0.002s sys 0.000s
I20260929 02:01:07.220155 17594 tablet_bootstrap.cc:492] T bbdc98a938b247bdaf113dc94c3b4d59 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:07.220584 17592 consensus_queue.cc:260] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.222064 17592 ts_tablet_manager.cc:1480] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20260929 02:01:07.222523 17592 tablet_bootstrap.cc:492] T a21bdeb06bb24a45945e00b68b17ca5b P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:07.225989 17594 tablet_bootstrap.cc:654] T bbdc98a938b247bdaf113dc94c3b4d59 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.227465 17584 ts_tablet_manager.cc:1962] T a3f4dfb792b946728e051ae1791cb75a P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:07.227697 17592 tablet_bootstrap.cc:654] T a21bdeb06bb24a45945e00b68b17ca5b P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.231070 16818 catalog_manager.cc:5838] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66). New cstate: current_term: 1 leader_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:07.234826 17595 tablet_bootstrap.cc:492] T ae7f6651c994403697a0f04508b70dae P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:07.235090 17595 ts_tablet_manager.cc:1449] T ae7f6651c994403697a0f04508b70dae P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.048s user 0.016s sys 0.001s
I20260929 02:01:07.236244 17595 raft_consensus.cc:359] T ae7f6651c994403697a0f04508b70dae P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.236603 17595 raft_consensus.cc:385] T ae7f6651c994403697a0f04508b70dae P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.236775 17595 raft_consensus.cc:740] T ae7f6651c994403697a0f04508b70dae P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.237020 17594 tablet_bootstrap.cc:492] T bbdc98a938b247bdaf113dc94c3b4d59 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:07.237257 17594 ts_tablet_manager.cc:1449] T bbdc98a938b247bdaf113dc94c3b4d59 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.017s user 0.003s sys 0.004s
I20260929 02:01:07.237176 17595 consensus_queue.cc:260] T ae7f6651c994403697a0f04508b70dae P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.238112 17594 raft_consensus.cc:359] T bbdc98a938b247bdaf113dc94c3b4d59 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.238471 17594 raft_consensus.cc:385] T bbdc98a938b247bdaf113dc94c3b4d59 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.238602 17594 raft_consensus.cc:740] T bbdc98a938b247bdaf113dc94c3b4d59 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.238946 17594 consensus_queue.cc:260] T bbdc98a938b247bdaf113dc94c3b4d59 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.243431 17594 ts_tablet_manager.cc:1480] T bbdc98a938b247bdaf113dc94c3b4d59 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260929 02:01:07.243891 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.243894 17595 ts_tablet_manager.cc:1480] T ae7f6651c994403697a0f04508b70dae P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.009s user 0.002s sys 0.000s
I20260929 02:01:07.244139 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.244349 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.244589 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.244844 17583 ts_tablet_manager.cc:1975] T 7f3e8d6962a940868e1ab2caa707cff1 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:07.244983 17583 log.cc:1199] T 7f3e8d6962a940868e1ab2caa707cff1 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/7f3e8d6962a940868e1ab2caa707cff1
I20260929 02:01:07.244952 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.245283 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.245493 17583 ts_tablet_manager.cc:1996] T 7f3e8d6962a940868e1ab2caa707cff1 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:07.245591 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.245872 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.246658 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 7f3e8d6962a940868e1ab2caa707cff1 (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:07.247292 17337 raft_consensus.cc:493] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:07.247454 17337 raft_consensus.cc:515] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.247750 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.247893 17595 tablet_bootstrap.cc:492] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:07.248049 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.248148 17337 leader_election.cc:290] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:07.248287 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.250144 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.250407 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.250589 17595 tablet_bootstrap.cc:654] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.250607 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.251611 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "167d937681dc4e509f1d5d4ce2358d27" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:07.251813 17117 raft_consensus.cc:2468] T 167d937681dc4e509f1d5d4ce2358d27 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:07.252352 17584 ts_tablet_manager.cc:1975] T a3f4dfb792b946728e051ae1791cb75a P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:07.252487 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "167d937681dc4e509f1d5d4ce2358d27" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:07.252638 17583 tablet_replica.cc:333] T a3f4dfb792b946728e051ae1791cb75a P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:07.252681 17253 raft_consensus.cc:2468] T 167d937681dc4e509f1d5d4ce2358d27 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:07.252851 17583 raft_consensus.cc:2243] T a3f4dfb792b946728e051ae1791cb75a P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:07.254000 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.212476
I20260929 02:01:07.254237 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.212476
I20260929 02:01:07.254458 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.212476
I20260929 02:01:07.254766 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.254958 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.253279 16919 leader_election.cc:304] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:07.255798 17337 raft_consensus.cc:2804] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:07.255901 17337 raft_consensus.cc:493] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:07.256001 17337 raft_consensus.cc:3060] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.256641 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:07.256934 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:07.257161 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:07.257444 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.257673 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.258996 17337 raft_consensus.cc:515] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.261166 17337 leader_election.cc:290] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Requested vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:07.261387 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "167d937681dc4e509f1d5d4ce2358d27" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:07.261608 17253 raft_consensus.cc:3060] T 167d937681dc4e509f1d5d4ce2358d27 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.262010 17592 tablet_bootstrap.cc:492] T a21bdeb06bb24a45945e00b68b17ca5b P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:07.262153 17592 ts_tablet_manager.cc:1449] T a21bdeb06bb24a45945e00b68b17ca5b P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.040s user 0.002s sys 0.001s
I20260929 02:01:07.262446 17583 raft_consensus.cc:2272] T a3f4dfb792b946728e051ae1791cb75a P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:07.263131 17583 ts_tablet_manager.cc:1962] T a3f4dfb792b946728e051ae1791cb75a P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:07.263161 17592 raft_consensus.cc:359] T a21bdeb06bb24a45945e00b68b17ca5b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.263504 17592 raft_consensus.cc:385] T a21bdeb06bb24a45945e00b68b17ca5b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.263649 17592 raft_consensus.cc:740] T a21bdeb06bb24a45945e00b68b17ca5b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.263996 17592 consensus_queue.cc:260] T a21bdeb06bb24a45945e00b68b17ca5b P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.264890 17592 ts_tablet_manager.cc:1480] T a21bdeb06bb24a45945e00b68b17ca5b P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260929 02:01:07.265403 17592 tablet_bootstrap.cc:492] T b402b4aee2a44330a87657f5a9469dc0 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:07.268296 17592 tablet_bootstrap.cc:654] T b402b4aee2a44330a87657f5a9469dc0 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.271096 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:07.271589 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.271857 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.023s user 0.004s sys 0.001s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":156,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:07.277818 17337 raft_consensus.cc:493] T 2e0146fb337d43b2921fb5fd34fb95ed P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:07.278024 17337 raft_consensus.cc:515] T 2e0146fb337d43b2921fb5fd34fb95ed P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.278748 17337 leader_election.cc:290] T 2e0146fb337d43b2921fb5fd34fb95ed P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:07.283047 17592 tablet_bootstrap.cc:492] T b402b4aee2a44330a87657f5a9469dc0 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:07.283398 17592 ts_tablet_manager.cc:1449] T b402b4aee2a44330a87657f5a9469dc0 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.018s user 0.010s sys 0.007s
I20260929 02:01:07.284623 17584 log.cc:1199] T a3f4dfb792b946728e051ae1791cb75a P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/a3f4dfb792b946728e051ae1791cb75a
I20260929 02:01:07.279814 17120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e0146fb337d43b2921fb5fd34fb95ed" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:07.261070 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "167d937681dc4e509f1d5d4ce2358d27" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:07.285372 17117 raft_consensus.cc:3060] T 167d937681dc4e509f1d5d4ce2358d27 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.285367 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.039s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":169,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
W20260929 02:01:07.285665 16919 leader_election.cc:343] T 2e0146fb337d43b2921fb5fd34fb95ed P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.285764 17253 raft_consensus.cc:2468] T 167d937681dc4e509f1d5d4ce2358d27 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:07.286201 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e0146fb337d43b2921fb5fd34fb95ed" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:07.286306 16921 leader_election.cc:304] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:07.284646 17592 raft_consensus.cc:359] T b402b4aee2a44330a87657f5a9469dc0 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
W20260929 02:01:07.286816 16921 leader_election.cc:343] T 2e0146fb337d43b2921fb5fd34fb95ed P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.287015 16921 leader_election.cc:304] T 2e0146fb337d43b2921fb5fd34fb95ed P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97df55a0361841319adfae27c9787b2f; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88
I20260929 02:01:07.287513 17337 raft_consensus.cc:2804] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:07.289016 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.032s user 0.007s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":996,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":55,"lbm_read_time_us":161,"lbm_reads_lt_1ms":8,"lbm_write_time_us":193,"lbm_writes_lt_1ms":4,"mutex_wait_us":6083,"reinsert_count":0,"update_count":18}
I20260929 02:01:07.294433 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.040s user 0.010s sys 0.002s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5401,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":834,"lbm_read_time_us":475,"lbm_reads_lt_1ms":27,"lbm_write_time_us":355,"lbm_writes_lt_1ms":9,"mutex_wait_us":23548,"peak_mem_usage":32768,"reinsert_count":0,"update_count":2}
I20260929 02:01:07.295815 17382 raft_consensus.cc:493] T c7892f83a6e34ba0899bb3ac41a749a5 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:07.296070 17382 raft_consensus.cc:515] T c7892f83a6e34ba0899bb3ac41a749a5 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.297015 17382 leader_election.cc:290] T c7892f83a6e34ba0899bb3ac41a749a5 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:07.298367 17592 raft_consensus.cc:385] T b402b4aee2a44330a87657f5a9469dc0 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.298305 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7892f83a6e34ba0899bb3ac41a749a5" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:07.298481 17592 raft_consensus.cc:740] T b402b4aee2a44330a87657f5a9469dc0 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.298770 17592 consensus_queue.cc:260] T b402b4aee2a44330a87657f5a9469dc0 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
W20260929 02:01:07.298979 17189 leader_election.cc:343] T c7892f83a6e34ba0899bb3ac41a749a5 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.299993 17595 tablet_bootstrap.cc:492] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:07.300211 17595 ts_tablet_manager.cc:1449] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.052s user 0.008s sys 0.019s
I20260929 02:01:07.301466 17595 raft_consensus.cc:359] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.301961 17595 raft_consensus.cc:385] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.302096 17595 raft_consensus.cc:740] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.302443 17595 consensus_queue.cc:260] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.303254 17625 raft_consensus.cc:2749] T 2e0146fb337d43b2921fb5fd34fb95ed P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:07.308889 17595 ts_tablet_manager.cc:1480] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.009s user 0.000s sys 0.004s
I20260929 02:01:07.309015 17592 ts_tablet_manager.cc:1480] T b402b4aee2a44330a87657f5a9469dc0 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.025s user 0.003s sys 0.000s
I20260929 02:01:07.309576 17592 tablet_bootstrap.cc:492] T 8f328eef30d541c7a61e7d7403e597c7 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:07.311728 17592 tablet_bootstrap.cc:654] T 8f328eef30d541c7a61e7d7403e597c7 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.313377 17595 tablet_bootstrap.cc:492] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:07.316099 17595 tablet_bootstrap.cc:654] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.319201 17584 ts_tablet_manager.cc:1996] T a3f4dfb792b946728e051ae1791cb75a P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:07.320796 17594 tablet_bootstrap.cc:492] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:07.322587 17583 ts_tablet_manager.cc:1975] T a3f4dfb792b946728e051ae1791cb75a P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:07.322664 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet a3f4dfb792b946728e051ae1791cb75a (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:07.322749 17583 log.cc:1199] T a3f4dfb792b946728e051ae1791cb75a P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/a3f4dfb792b946728e051ae1791cb75a
I20260929 02:01:07.323194 17583 ts_tablet_manager.cc:1996] T a3f4dfb792b946728e051ae1791cb75a P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:07.324617 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet a3f4dfb792b946728e051ae1791cb75a (table table-1 [id=8e98daa65ee946cbb84751b44e2cf2f4]) successfully deleted
I20260929 02:01:07.326066 17594 tablet_bootstrap.cc:654] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.327469 17595 tablet_bootstrap.cc:492] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:07.326337 17120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7892f83a6e34ba0899bb3ac41a749a5" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:07.327665 17595 ts_tablet_manager.cc:1449] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.004s
I20260929 02:01:07.326522 17117 raft_consensus.cc:2468] T 167d937681dc4e509f1d5d4ce2358d27 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
W20260929 02:01:07.328159 17186 leader_election.cc:343] T c7892f83a6e34ba0899bb3ac41a749a5 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.328346 17186 leader_election.cc:304] T c7892f83a6e34ba0899bb3ac41a749a5 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f
I20260929 02:01:07.329159 17382 raft_consensus.cc:2749] T c7892f83a6e34ba0899bb3ac41a749a5 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:07.330163 17595 raft_consensus.cc:359] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.330439 17595 raft_consensus.cc:385] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.331059 17595 raft_consensus.cc:740] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.331430 17595 consensus_queue.cc:260] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
W20260929 02:01:07.334417 16921 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.RequestConsensusVote -> {remote=127.0.71.67:44233, user_credentials={real_user=slave}} blocked reactor thread for 47612.7us
I20260929 02:01:07.335681 17337 raft_consensus.cc:697] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Becoming Leader. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Running, Role: LEADER
I20260929 02:01:07.336099 17595 ts_tablet_manager.cc:1480] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.008s user 0.001s sys 0.001s
I20260929 02:01:07.336575 17337 consensus_queue.cc:237] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.338164 17592 tablet_bootstrap.cc:492] T 8f328eef30d541c7a61e7d7403e597c7 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:07.338569 17592 ts_tablet_manager.cc:1449] T 8f328eef30d541c7a61e7d7403e597c7 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.029s user 0.002s sys 0.002s
I20260929 02:01:07.339629 17595 tablet_bootstrap.cc:492] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:07.339605 17592 raft_consensus.cc:359] T 8f328eef30d541c7a61e7d7403e597c7 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.339937 17592 raft_consensus.cc:385] T 8f328eef30d541c7a61e7d7403e597c7 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.340093 17592 raft_consensus.cc:740] T 8f328eef30d541c7a61e7d7403e597c7 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.340471 17592 consensus_queue.cc:260] T 8f328eef30d541c7a61e7d7403e597c7 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.341154 17592 ts_tablet_manager.cc:1480] T 8f328eef30d541c7a61e7d7403e597c7 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 02:01:07.341627 17592 tablet_bootstrap.cc:492] T 306f1984e9324a949db3435c56fcfb8f P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:07.341655 17594 tablet_bootstrap.cc:492] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:07.341812 17594 ts_tablet_manager.cc:1449] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.048s user 0.001s sys 0.007s
I20260929 02:01:07.342730 17594 raft_consensus.cc:359] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.342952 17594 raft_consensus.cc:385] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.343060 17594 raft_consensus.cc:740] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.343353 17594 consensus_queue.cc:260] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.343896 17594 ts_tablet_manager.cc:1480] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 02:01:07.345193 17594 tablet_bootstrap.cc:492] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:07.346125 17595 tablet_bootstrap.cc:654] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.349099 17592 tablet_bootstrap.cc:654] T 306f1984e9324a949db3435c56fcfb8f P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.350612 17624 raft_consensus.cc:493] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:07.350692 17594 tablet_bootstrap.cc:654] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.350786 17624 raft_consensus.cc:515] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.352273 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b2b8f6f05b74c849d19705a2e8ab0e7" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:07.352576 17117 raft_consensus.cc:2468] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:07.353282 16919 leader_election.cc:304] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:07.353749 17623 raft_consensus.cc:2804] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:07.353935 17623 raft_consensus.cc:493] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:07.354169 17623 raft_consensus.cc:3060] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.356300 17595 tablet_bootstrap.cc:492] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:07.356501 17595 ts_tablet_manager.cc:1449] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.017s user 0.004s sys 0.001s
I20260929 02:01:07.357868 17595 raft_consensus.cc:359] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.358120 17595 raft_consensus.cc:385] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.358213 17595 raft_consensus.cc:740] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.358269 17624 leader_election.cc:290] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:07.358556 17595 consensus_queue.cc:260] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.359225 17595 ts_tablet_manager.cc:1480] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 02:01:07.359601 17595 tablet_bootstrap.cc:492] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:07.360559 17623 raft_consensus.cc:515] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.361974 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b2b8f6f05b74c849d19705a2e8ab0e7" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:07.362190 17117 raft_consensus.cc:3060] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.364787 17623 leader_election.cc:290] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Requested vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:07.366444 17117 raft_consensus.cc:2468] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:07.367296 17595 tablet_bootstrap.cc:654] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.367137 16919 leader_election.cc:304] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:07.368057 17624 raft_consensus.cc:2804] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:07.368053 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b2b8f6f05b74c849d19705a2e8ab0e7" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:07.368188 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5b2b8f6f05b74c849d19705a2e8ab0e7" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:07.368281 17624 raft_consensus.cc:697] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Becoming Leader. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Running, Role: LEADER
I20260929 02:01:07.368299 17251 raft_consensus.cc:2468] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:07.368592 17624 consensus_queue.cc:237] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.378186 17595 tablet_bootstrap.cc:492] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:07.378389 17595 ts_tablet_manager.cc:1449] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.019s user 0.001s sys 0.008s
I20260929 02:01:07.381367 17592 tablet_bootstrap.cc:492] T 306f1984e9324a949db3435c56fcfb8f P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:07.381032 16817 catalog_manager.cc:5838] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97df55a0361841319adfae27c9787b2f (127.0.71.65). New cstate: current_term: 1 leader_uuid: "97df55a0361841319adfae27c9787b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:07.381752 17592 ts_tablet_manager.cc:1449] T 306f1984e9324a949db3435c56fcfb8f P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.040s user 0.000s sys 0.005s
I20260929 02:01:07.382666 17592 raft_consensus.cc:359] T 306f1984e9324a949db3435c56fcfb8f P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.382987 17592 raft_consensus.cc:385] T 306f1984e9324a949db3435c56fcfb8f P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.383113 17592 raft_consensus.cc:740] T 306f1984e9324a949db3435c56fcfb8f P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.383466 17592 consensus_queue.cc:260] T 306f1984e9324a949db3435c56fcfb8f P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.384193 17592 ts_tablet_manager.cc:1480] T 306f1984e9324a949db3435c56fcfb8f P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 02:01:07.384611 17592 tablet_bootstrap.cc:492] T 05f61c0da0dd4a20a60cc2b3390cb07c P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:07.386876 17594 tablet_bootstrap.cc:492] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:07.387161 17594 ts_tablet_manager.cc:1449] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.042s user 0.010s sys 0.006s
I20260929 02:01:07.387969 17592 tablet_bootstrap.cc:654] T 05f61c0da0dd4a20a60cc2b3390cb07c P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.388994 17594 raft_consensus.cc:359] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.390652 17594 raft_consensus.cc:385] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.390796 17594 raft_consensus.cc:740] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.391130 17594 consensus_queue.cc:260] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.392143 17594 ts_tablet_manager.cc:1480] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 02:01:07.392550 17594 tablet_bootstrap.cc:492] T 7239b23c6e30483d9bd0738e07c63610 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:07.394927 17594 tablet_bootstrap.cc:654] T 7239b23c6e30483d9bd0738e07c63610 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.395310 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.395649 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.395933 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.396214 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.398358 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.398658 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.398865 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.401805 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.100535
I20260929 02:01:07.402104 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.100535
I20260929 02:01:07.402419 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.100535
I20260929 02:01:07.402850 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.403121 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.410079 17594 tablet_bootstrap.cc:492] T 7239b23c6e30483d9bd0738e07c63610 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:07.410244 17594 ts_tablet_manager.cc:1449] T 7239b23c6e30483d9bd0738e07c63610 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.018s user 0.007s sys 0.000s
I20260929 02:01:07.411549 17594 raft_consensus.cc:359] T 7239b23c6e30483d9bd0738e07c63610 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.411842 17594 raft_consensus.cc:385] T 7239b23c6e30483d9bd0738e07c63610 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.412531 17594 raft_consensus.cc:740] T 7239b23c6e30483d9bd0738e07c63610 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.412477 16818 catalog_manager.cc:5838] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97df55a0361841319adfae27c9787b2f (127.0.71.65). New cstate: current_term: 1 leader_uuid: "97df55a0361841319adfae27c9787b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:07.412900 17594 consensus_queue.cc:260] T 7239b23c6e30483d9bd0738e07c63610 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.413426 17594 ts_tablet_manager.cc:1480] T 7239b23c6e30483d9bd0738e07c63610 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.003s user 0.002s sys 0.001s
I20260929 02:01:07.413774 17594 tablet_bootstrap.cc:492] T ae7f6651c994403697a0f04508b70dae P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:07.415591 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.415961 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.416285 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.416587 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.419384 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.419687 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.419931 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.422749 17594 tablet_bootstrap.cc:654] T ae7f6651c994403697a0f04508b70dae P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.432602 17382 raft_consensus.cc:493] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:07.432897 17382 raft_consensus.cc:515] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.433862 17382 leader_election.cc:290] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:07.435164 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f328eef30d541c7a61e7d7403e597c7" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:07.435456 16986 raft_consensus.cc:2468] T 8f328eef30d541c7a61e7d7403e597c7 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 0.
I20260929 02:01:07.436353 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f328eef30d541c7a61e7d7403e597c7" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:07.437332 17189 leader_election.cc:304] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
W20260929 02:01:07.438059 17186 leader_election.cc:343] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.438598 17382 raft_consensus.cc:2804] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:07.438722 17382 raft_consensus.cc:493] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:07.438853 17382 raft_consensus.cc:3060] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.443818 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:07.444538 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.445703 17337 raft_consensus.cc:493] T a21bdeb06bb24a45945e00b68b17ca5b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:07.445945 17337 raft_consensus.cc:515] T a21bdeb06bb24a45945e00b68b17ca5b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.446833 17337 leader_election.cc:290] T a21bdeb06bb24a45945e00b68b17ca5b P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:07.447686 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a21bdeb06bb24a45945e00b68b17ca5b" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
W20260929 02:01:07.448516 16921 leader_election.cc:343] T a21bdeb06bb24a45945e00b68b17ca5b P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.448932 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a21bdeb06bb24a45945e00b68b17ca5b" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
W20260929 02:01:07.449726 16919 leader_election.cc:343] T a21bdeb06bb24a45945e00b68b17ca5b P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.449896 16919 leader_election.cc:304] T a21bdeb06bb24a45945e00b68b17ca5b P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97df55a0361841319adfae27c9787b2f; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88
I20260929 02:01:07.450531 17337 raft_consensus.cc:2749] T a21bdeb06bb24a45945e00b68b17ca5b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:07.454445 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.454895 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.455914 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.456547 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.457167 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.457991 17337 raft_consensus.cc:493] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:07.458189 17337 raft_consensus.cc:515] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.458945 17337 leader_election.cc:290] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:07.459846 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae7f6651c994403697a0f04508b70dae" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:07.460803 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae7f6651c994403697a0f04508b70dae" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:07.461063 17251 raft_consensus.cc:2468] T ae7f6651c994403697a0f04508b70dae P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
W20260929 02:01:07.461079 16919 leader_election.cc:343] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.461567 16921 leader_election.cc:304] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc
I20260929 02:01:07.462144 17337 raft_consensus.cc:2804] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:07.462248 17337 raft_consensus.cc:493] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:07.462356 17337 raft_consensus.cc:3060] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.466109 17337 raft_consensus.cc:515] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.467553 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae7f6651c994403697a0f04508b70dae" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
W20260929 02:01:07.468313 16919 leader_election.cc:343] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.468699 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.012s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":11692}
I20260929 02:01:07.469092 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.012s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":11531}
I20260929 02:01:07.469331 17337 leader_election.cc:290] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Requested vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:07.469359 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae7f6651c994403697a0f04508b70dae" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:07.469318 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.013s user 0.002s sys 0.005s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":196,"lbm_writes_lt_1ms":4,"mutex_wait_us":5022,"reinsert_count":0,"update_count":1}
I20260929 02:01:07.469573 17251 raft_consensus.cc:3060] T ae7f6651c994403697a0f04508b70dae P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.470106 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.152528
I20260929 02:01:07.470463 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.152528
I20260929 02:01:07.473730 17251 raft_consensus.cc:2468] T ae7f6651c994403697a0f04508b70dae P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:07.474313 16921 leader_election.cc:304] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc
I20260929 02:01:07.474859 17337 raft_consensus.cc:2804] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:07.475039 17337 raft_consensus.cc:697] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Becoming Leader. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Running, Role: LEADER
I20260929 02:01:07.475342 17337 consensus_queue.cc:237] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.480191 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:07.480537 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.480831 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.488526 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.092s user 0.006s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":213,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:07.489444 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.490152 17382 raft_consensus.cc:515] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.491914 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f328eef30d541c7a61e7d7403e597c7" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:07.492213 16986 raft_consensus.cc:3060] T 8f328eef30d541c7a61e7d7403e597c7 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.500126 17382 leader_election.cc:290] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:07.500556 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f328eef30d541c7a61e7d7403e597c7" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:07.501046 16986 raft_consensus.cc:2468] T 8f328eef30d541c7a61e7d7403e597c7 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 1.
I20260929 02:01:07.501693 17189 leader_election.cc:304] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
W20260929 02:01:07.505033 17186 leader_election.cc:343] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.507766 17595 raft_consensus.cc:359] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.508045 17595 raft_consensus.cc:385] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.508167 17595 raft_consensus.cc:740] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.508564 17595 consensus_queue.cc:260] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.509074 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.092s user 0.007s sys 0.001s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":208,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:07.509981 17382 raft_consensus.cc:2804] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:07.521844 17595 ts_tablet_manager.cc:1480] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.143s user 0.003s sys 0.003s
I20260929 02:01:07.522320 17595 tablet_bootstrap.cc:492] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:07.524806 17398 raft_consensus.cc:493] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:07.525113 17398 raft_consensus.cc:515] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.526173 17398 leader_election.cc:290] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:07.527511 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6cac2e61f6f4795869af850549096a9" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:07.527755 17251 raft_consensus.cc:2468] T c6cac2e61f6f4795869af850549096a9 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:07.528509 17055 leader_election.cc:304] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:07.528560 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6cac2e61f6f4795869af850549096a9" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:07.529053 17398 raft_consensus.cc:2804] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:07.529189 17398 raft_consensus.cc:493] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260929 02:01:07.529261 17056 leader_election.cc:343] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.529359 17398 raft_consensus.cc:3060] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.534128 17398 raft_consensus.cc:515] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.535871 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6cac2e61f6f4795869af850549096a9" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
W20260929 02:01:07.536684 17056 leader_election.cc:343] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.538029 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6cac2e61f6f4795869af850549096a9" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:07.538370 17251 raft_consensus.cc:3060] T c6cac2e61f6f4795869af850549096a9 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.543396 17251 raft_consensus.cc:2468] T c6cac2e61f6f4795869af850549096a9 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:07.544152 17055 leader_election.cc:304] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters: 97df55a0361841319adfae27c9787b2f
I20260929 02:01:07.544631 17543 raft_consensus.cc:2804] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:07.544826 17543 raft_consensus.cc:697] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Becoming Leader. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Running, Role: LEADER
I20260929 02:01:07.545233 17543 consensus_queue.cc:237] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.549682 17398 leader_election.cc:290] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:07.550204 17382 raft_consensus.cc:697] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Becoming Leader. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Running, Role: LEADER
I20260929 02:01:07.550617 17382 consensus_queue.cc:237] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.552183 17640 raft_consensus.cc:493] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:07.552496 17640 raft_consensus.cc:515] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.553952 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5acedab1b27c414fa5181e35aa9ac24b" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
W20260929 02:01:07.554808 17189 leader_election.cc:343] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.555537 17640 leader_election.cc:290] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:07.556434 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5acedab1b27c414fa5181e35aa9ac24b" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:07.556772 17117 raft_consensus.cc:2468] T 5acedab1b27c414fa5181e35aa9ac24b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 0.
I20260929 02:01:07.557525 17186 leader_election.cc:304] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters: 97df55a0361841319adfae27c9787b2f
I20260929 02:01:07.558202 17640 raft_consensus.cc:2804] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:07.558420 17640 raft_consensus.cc:493] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:07.558616 17640 raft_consensus.cc:3060] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.559774 17398 raft_consensus.cc:493] T bbdc98a938b247bdaf113dc94c3b4d59 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:07.560050 17398 raft_consensus.cc:515] T bbdc98a938b247bdaf113dc94c3b4d59 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.560262 16817 catalog_manager.cc:5838] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66). New cstate: current_term: 1 leader_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:07.561021 17398 leader_election.cc:290] T bbdc98a938b247bdaf113dc94c3b4d59 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:07.564595 17640 raft_consensus.cc:515] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.571431 17640 leader_election.cc:290] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:07.573148 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5acedab1b27c414fa5181e35aa9ac24b" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
W20260929 02:01:07.574014 17189 leader_election.cc:343] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.574991 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5acedab1b27c414fa5181e35aa9ac24b" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:07.575286 17117 raft_consensus.cc:3060] T 5acedab1b27c414fa5181e35aa9ac24b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.579493 17117 raft_consensus.cc:2468] T 5acedab1b27c414fa5181e35aa9ac24b P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 1.
I20260929 02:01:07.580487 17186 leader_election.cc:304] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters: 97df55a0361841319adfae27c9787b2f
I20260929 02:01:07.580960 17640 raft_consensus.cc:2804] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:07.581246 17640 raft_consensus.cc:697] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Becoming Leader. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Running, Role: LEADER
I20260929 02:01:07.581700 17640 consensus_queue.cc:237] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.584996 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbdc98a938b247bdaf113dc94c3b4d59" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
W20260929 02:01:07.585896 17056 leader_election.cc:343] T bbdc98a938b247bdaf113dc94c3b4d59 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.588047 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbdc98a938b247bdaf113dc94c3b4d59" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
W20260929 02:01:07.588959 17055 leader_election.cc:343] T bbdc98a938b247bdaf113dc94c3b4d59 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.589133 17055 leader_election.cc:304] T bbdc98a938b247bdaf113dc94c3b4d59 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc; no voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88
I20260929 02:01:07.589542 17398 raft_consensus.cc:2749] T bbdc98a938b247bdaf113dc94c3b4d59 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:07.590070 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.590596 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.591037 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.593690 17398 consensus_queue.cc:1048] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:07.600476 17624 raft_consensus.cc:493] T 306f1984e9324a949db3435c56fcfb8f P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:07.600725 17624 raft_consensus.cc:515] T 306f1984e9324a949db3435c56fcfb8f P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.601683 17624 leader_election.cc:290] T 306f1984e9324a949db3435c56fcfb8f P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:07.602160 16817 catalog_manager.cc:5838] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88 reported cstate change: term changed from 0 to 1, leader changed from <none> to d103dd4823044faebe46100a0dfbee88 (127.0.71.67). New cstate: current_term: 1 leader_uuid: "d103dd4823044faebe46100a0dfbee88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: HEALTHY } } }
I20260929 02:01:07.604270 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306f1984e9324a949db3435c56fcfb8f" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
W20260929 02:01:07.604964 16919 leader_election.cc:343] T 306f1984e9324a949db3435c56fcfb8f P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.615250 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306f1984e9324a949db3435c56fcfb8f" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
W20260929 02:01:07.616155 16921 leader_election.cc:343] T 306f1984e9324a949db3435c56fcfb8f P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.616454 16921 leader_election.cc:304] T 306f1984e9324a949db3435c56fcfb8f P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97df55a0361841319adfae27c9787b2f; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88
I20260929 02:01:07.617098 17624 raft_consensus.cc:2749] T 306f1984e9324a949db3435c56fcfb8f P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:07.648001 17595 tablet_bootstrap.cc:700] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88: Time spent opening tablet: real 0.124s user 0.002s sys 0.000s
I20260929 02:01:07.648614 17595 tablet_bootstrap.cc:654] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.653786 17543 consensus_queue.cc:1048] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:07.659600 17639 raft_consensus.cc:493] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:07.660163 17639 raft_consensus.cc:515] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.662082 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e0146fb337d43b2921fb5fd34fb95ed" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:07.662655 16986 raft_consensus.cc:2468] T 2e0146fb337d43b2921fb5fd34fb95ed P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 0.
I20260929 02:01:07.663353 17189 leader_election.cc:304] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:07.664147 17382 raft_consensus.cc:2804] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:07.664539 17382 raft_consensus.cc:493] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:07.664916 17382 raft_consensus.cc:3060] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.666051 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e0146fb337d43b2921fb5fd34fb95ed" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
W20260929 02:01:07.666908 17186 leader_election.cc:343] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.668282 17639 leader_election.cc:290] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:07.676239 17592 tablet_bootstrap.cc:492] T 05f61c0da0dd4a20a60cc2b3390cb07c P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:07.676575 17594 tablet_bootstrap.cc:492] T ae7f6651c994403697a0f04508b70dae P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:07.677409 17594 ts_tablet_manager.cc:1449] T ae7f6651c994403697a0f04508b70dae P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.264s user 0.100s sys 0.145s
I20260929 02:01:07.678212 17592 ts_tablet_manager.cc:1449] T 05f61c0da0dd4a20a60cc2b3390cb07c P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.294s user 0.045s sys 0.078s
I20260929 02:01:07.678313 17594 raft_consensus.cc:359] T ae7f6651c994403697a0f04508b70dae P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.678536 17594 raft_consensus.cc:385] T ae7f6651c994403697a0f04508b70dae P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.678617 17594 raft_consensus.cc:740] T ae7f6651c994403697a0f04508b70dae P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.678819 16817 catalog_manager.cc:5838] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97df55a0361841319adfae27c9787b2f (127.0.71.65). New cstate: current_term: 1 leader_uuid: "97df55a0361841319adfae27c9787b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:07.678946 17594 consensus_queue.cc:260] T ae7f6651c994403697a0f04508b70dae P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.679903 17594 ts_tablet_manager.cc:1480] T ae7f6651c994403697a0f04508b70dae P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:01:07.680200 17594 tablet_bootstrap.cc:492] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:07.681995 17592 raft_consensus.cc:359] T 05f61c0da0dd4a20a60cc2b3390cb07c P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.682188 17594 tablet_bootstrap.cc:654] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.682344 17592 raft_consensus.cc:385] T 05f61c0da0dd4a20a60cc2b3390cb07c P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.682510 17592 raft_consensus.cc:740] T 05f61c0da0dd4a20a60cc2b3390cb07c P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.682863 17592 consensus_queue.cc:260] T 05f61c0da0dd4a20a60cc2b3390cb07c P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.683521 17592 ts_tablet_manager.cc:1480] T 05f61c0da0dd4a20a60cc2b3390cb07c P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 02:01:07.684088 17592 tablet_bootstrap.cc:492] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:07.686340 17592 tablet_bootstrap.cc:654] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.697018 17543 consensus_queue.cc:1048] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:07.710291 17337 raft_consensus.cc:493] T b402b4aee2a44330a87657f5a9469dc0 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:07.710558 17337 raft_consensus.cc:515] T b402b4aee2a44330a87657f5a9469dc0 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.712385 17337 leader_election.cc:290] T b402b4aee2a44330a87657f5a9469dc0 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:07.712790 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b402b4aee2a44330a87657f5a9469dc0" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
W20260929 02:01:07.713668 16919 leader_election.cc:343] T b402b4aee2a44330a87657f5a9469dc0 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.716964 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b402b4aee2a44330a87657f5a9469dc0" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
W20260929 02:01:07.717679 16921 leader_election.cc:343] T b402b4aee2a44330a87657f5a9469dc0 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.717865 16921 leader_election.cc:304] T b402b4aee2a44330a87657f5a9469dc0 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97df55a0361841319adfae27c9787b2f; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88
I20260929 02:01:07.718637 17337 raft_consensus.cc:2749] T b402b4aee2a44330a87657f5a9469dc0 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:07.720516 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.250s user 0.012s sys 0.003s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5475,"cfile_init":6,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":158648,"dirs.run_cpu_time_us":204,"dirs.run_wall_time_us":2882,"lbm_read_time_us":778,"lbm_reads_lt_1ms":31,"lbm_write_time_us":766,"lbm_writes_lt_1ms":9,"peak_mem_usage":32768,"reinsert_count":0,"update_count":2}
I20260929 02:01:07.721812 17543 consensus_queue.cc:1048] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:07.722532 17382 raft_consensus.cc:515] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.725615 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.130s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":129777}
I20260929 02:01:07.726001 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.130s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":130096}
I20260929 02:01:07.726608 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.135s user 0.006s sys 0.000s Metrics: {"bytes_written":493,"delete_count":0,"lbm_write_time_us":187,"lbm_writes_lt_1ms":4,"mutex_wait_us":126440,"reinsert_count":0,"update_count":3}
I20260929 02:01:07.727245 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e0146fb337d43b2921fb5fd34fb95ed" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
W20260929 02:01:07.728047 17186 leader_election.cc:343] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.728375 17382 leader_election.cc:290] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:07.728685 17543 raft_consensus.cc:493] T ae7f6651c994403697a0f04508b70dae P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:07.728822 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e0146fb337d43b2921fb5fd34fb95ed" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:07.728894 17543 raft_consensus.cc:515] T ae7f6651c994403697a0f04508b70dae P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.729090 16986 raft_consensus.cc:3060] T 2e0146fb337d43b2921fb5fd34fb95ed P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.729715 17543 leader_election.cc:290] T ae7f6651c994403697a0f04508b70dae P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:07.731092 17252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae7f6651c994403697a0f04508b70dae" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:07.731328 17252 raft_consensus.cc:2393] T ae7f6651c994403697a0f04508b70dae P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8668afa5b5ca4ebdabc812d0789de7bc in current term 1: Already voted for candidate 97df55a0361841319adfae27c9787b2f in this term.
I20260929 02:01:07.731328 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "ae7f6651c994403697a0f04508b70dae" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:07.733659 17595 tablet_bootstrap.cc:492] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:07.733628 17056 leader_election.cc:304] T ae7f6651c994403697a0f04508b70dae P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc; no voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88
I20260929 02:01:07.733834 17595 ts_tablet_manager.cc:1449] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.212s user 0.002s sys 0.003s
I20260929 02:01:07.734263 17543 raft_consensus.cc:3060] T ae7f6651c994403697a0f04508b70dae P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.735127 17595 raft_consensus.cc:359] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.735357 17595 raft_consensus.cc:385] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.735466 17595 raft_consensus.cc:740] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.735806 17595 consensus_queue.cc:260] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.736475 17595 ts_tablet_manager.cc:1480] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 02:01:07.736775 17595 tablet_bootstrap.cc:492] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:07.737881 17543 raft_consensus.cc:2749] T ae7f6651c994403697a0f04508b70dae P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:07.745077 16817 catalog_manager.cc:5838] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88 reported cstate change: term changed from 0 to 1, leader changed from <none> to d103dd4823044faebe46100a0dfbee88 (127.0.71.67). New cstate: current_term: 1 leader_uuid: "d103dd4823044faebe46100a0dfbee88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: HEALTHY } } }
I20260929 02:01:07.749246 17369 raft_consensus.cc:493] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:07.749432 17369 raft_consensus.cc:515] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.750705 17369 leader_election.cc:290] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:07.750672 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7892f83a6e34ba0899bb3ac41a749a5" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:07.750990 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7892f83a6e34ba0899bb3ac41a749a5" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:07.751215 17253 raft_consensus.cc:2468] T c7892f83a6e34ba0899bb3ac41a749a5 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
W20260929 02:01:07.751260 17056 leader_election.cc:343] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.751641 17055 leader_election.cc:304] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters: 97df55a0361841319adfae27c9787b2f
I20260929 02:01:07.752044 17369 raft_consensus.cc:2804] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:07.752147 17369 raft_consensus.cc:493] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:07.752249 17369 raft_consensus.cc:3060] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.752475 17592 tablet_bootstrap.cc:492] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:07.752656 17592 ts_tablet_manager.cc:1449] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.069s user 0.006s sys 0.000s
I20260929 02:01:07.753641 17592 raft_consensus.cc:359] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.753968 17592 raft_consensus.cc:385] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.754117 17592 raft_consensus.cc:740] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.754549 17592 consensus_queue.cc:260] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.756225 17369 raft_consensus.cc:515] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.757782 17639 raft_consensus.cc:493] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:07.757964 17639 raft_consensus.cc:515] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.758178 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7892f83a6e34ba0899bb3ac41a749a5" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:07.758212 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.758349 17253 raft_consensus.cc:3060] T c7892f83a6e34ba0899bb3ac41a749a5 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.758605 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.758944 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.759462 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05f61c0da0dd4a20a60cc2b3390cb07c" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:07.759594 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05f61c0da0dd4a20a60cc2b3390cb07c" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:07.759681 16984 raft_consensus.cc:2468] T 05f61c0da0dd4a20a60cc2b3390cb07c P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 0.
I20260929 02:01:07.760200 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7892f83a6e34ba0899bb3ac41a749a5" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
W20260929 02:01:07.760797 17056 leader_election.cc:343] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.758630 17639 leader_election.cc:290] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:07.760682 17189 leader_election.cc:304] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:07.761938 17639 raft_consensus.cc:2804] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:07.762056 17639 raft_consensus.cc:493] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:07.762174 17639 raft_consensus.cc:3060] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.762509 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.762705 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.762866 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.763120 17592 ts_tablet_manager.cc:1480] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20260929 02:01:07.763190 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.216888
I20260929 02:01:07.763487 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.216888
I20260929 02:01:07.763752 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.764169 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
W20260929 02:01:07.764681 17186 leader_election.cc:343] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.766383 17592 tablet_bootstrap.cc:492] T c6cac2e61f6f4795869af850549096a9 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:07.768537 17592 tablet_bootstrap.cc:654] T c6cac2e61f6f4795869af850549096a9 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.768869 17639 raft_consensus.cc:515] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.770153 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05f61c0da0dd4a20a60cc2b3390cb07c" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:07.770435 16983 raft_consensus.cc:3060] T 05f61c0da0dd4a20a60cc2b3390cb07c P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.771740 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05f61c0da0dd4a20a60cc2b3390cb07c" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
W20260929 02:01:07.772590 17186 leader_election.cc:343] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.772887 17639 leader_election.cc:290] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:07.774767 17369 leader_election.cc:290] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:07.777256 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.017s user 0.001s sys 0.004s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":194,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:07.780157 16983 raft_consensus.cc:2468] T 05f61c0da0dd4a20a60cc2b3390cb07c P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 1.
I20260929 02:01:07.780944 17189 leader_election.cc:304] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc
I20260929 02:01:07.781734 17639 raft_consensus.cc:2804] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:07.781997 17639 raft_consensus.cc:697] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Becoming Leader. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Running, Role: LEADER
I20260929 02:01:07.782756 17639 consensus_queue.cc:237] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.783685 17592 tablet_bootstrap.cc:492] T c6cac2e61f6f4795869af850549096a9 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:07.783941 17592 ts_tablet_manager.cc:1449] T c6cac2e61f6f4795869af850549096a9 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.018s user 0.001s sys 0.005s
I20260929 02:01:07.785121 17592 raft_consensus.cc:359] T c6cac2e61f6f4795869af850549096a9 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.785503 17592 raft_consensus.cc:385] T c6cac2e61f6f4795869af850549096a9 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.785682 17592 raft_consensus.cc:740] T c6cac2e61f6f4795869af850549096a9 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.786078 17592 consensus_queue.cc:260] T c6cac2e61f6f4795869af850549096a9 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.790481 17592 ts_tablet_manager.cc:1480] T c6cac2e61f6f4795869af850549096a9 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260929 02:01:07.791877 17337 raft_consensus.cc:493] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:07.792062 17337 raft_consensus.cc:515] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.792994 17337 leader_election.cc:290] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:07.793803 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7239b23c6e30483d9bd0738e07c63610" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:07.793900 17592 tablet_bootstrap.cc:492] T 5acedab1b27c414fa5181e35aa9ac24b P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
W20260929 02:01:07.794415 16921 leader_election.cc:343] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.794637 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7239b23c6e30483d9bd0738e07c63610" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:07.795256 17117 raft_consensus.cc:2468] T 7239b23c6e30483d9bd0738e07c63610 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:07.796262 17592 tablet_bootstrap.cc:654] T 5acedab1b27c414fa5181e35aa9ac24b P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.796187 16919 leader_election.cc:304] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters: d103dd4823044faebe46100a0dfbee88
I20260929 02:01:07.797700 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.034s user 0.012s sys 0.000s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5606,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":85,"lbm_read_time_us":559,"lbm_reads_lt_1ms":27,"lbm_write_time_us":335,"lbm_writes_lt_1ms":9,"mutex_wait_us":4065,"peak_mem_usage":32768,"reinsert_count":0,"update_count":4}
I20260929 02:01:07.798735 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.343s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":343234}
I20260929 02:01:07.799094 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.292s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":291673}
I20260929 02:01:07.801158 17337 raft_consensus.cc:2804] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:07.801328 17337 raft_consensus.cc:493] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:07.801473 17337 raft_consensus.cc:3060] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.804716 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.349s user 0.003s sys 0.004s Metrics: {"bytes_written":552,"delete_count":0,"lbm_write_time_us":178,"lbm_writes_lt_1ms":4,"mutex_wait_us":329412,"reinsert_count":0,"update_count":5}
I20260929 02:01:07.806540 17337 raft_consensus.cc:515] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.808086 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7239b23c6e30483d9bd0738e07c63610" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:07.808369 17117 raft_consensus.cc:3060] T 7239b23c6e30483d9bd0738e07c63610 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.809719 17592 tablet_bootstrap.cc:492] T 5acedab1b27c414fa5181e35aa9ac24b P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:07.809911 17592 ts_tablet_manager.cc:1449] T 5acedab1b27c414fa5181e35aa9ac24b P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.016s user 0.004s sys 0.002s
I20260929 02:01:07.810726 17592 raft_consensus.cc:359] T 5acedab1b27c414fa5181e35aa9ac24b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.810947 17592 raft_consensus.cc:385] T 5acedab1b27c414fa5181e35aa9ac24b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.811101 17592 raft_consensus.cc:740] T 5acedab1b27c414fa5181e35aa9ac24b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.811403 17592 consensus_queue.cc:260] T 5acedab1b27c414fa5181e35aa9ac24b P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.811777 17624 raft_consensus.cc:493] T c6cac2e61f6f4795869af850549096a9 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:07.811920 17624 raft_consensus.cc:515] T c6cac2e61f6f4795869af850549096a9 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.812583 17624 leader_election.cc:290] T c6cac2e61f6f4795869af850549096a9 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:07.812573 17117 raft_consensus.cc:2468] T 7239b23c6e30483d9bd0738e07c63610 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:07.812966 17337 leader_election.cc:290] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Requested vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:07.813167 16919 leader_election.cc:304] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:07.813585 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6cac2e61f6f4795869af850549096a9" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:07.813517 17592 ts_tablet_manager.cc:1480] T 5acedab1b27c414fa5181e35aa9ac24b P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260929 02:01:07.813616 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c6cac2e61f6f4795869af850549096a9" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:07.813964 17251 raft_consensus.cc:2393] T c6cac2e61f6f4795869af850549096a9 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97df55a0361841319adfae27c9787b2f in current term 1: Already voted for candidate 8668afa5b5ca4ebdabc812d0789de7bc in this term.
I20260929 02:01:07.814420 17592 tablet_bootstrap.cc:492] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:07.814468 17252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7239b23c6e30483d9bd0738e07c63610" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
W20260929 02:01:07.815412 16921 leader_election.cc:343] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.816229 17624 raft_consensus.cc:2804] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:07.816404 17624 raft_consensus.cc:697] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Becoming Leader. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Running, Role: LEADER
I20260929 02:01:07.816697 17624 consensus_queue.cc:237] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.817327 17592 tablet_bootstrap.cc:654] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.818601 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.416s user 0.018s sys 0.000s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":14194,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":94726,"dirs.run_cpu_time_us":303,"dirs.run_wall_time_us":1762,"lbm_read_time_us":864,"lbm_reads_lt_1ms":23,"lbm_write_time_us":488,"lbm_writes_lt_1ms":9,"mutex_wait_us":79033,"peak_mem_usage":32768,"reinsert_count":0,"thread_start_us":44712,"threads_started":1,"update_count":19}
I20260929 02:01:07.819377 16986 raft_consensus.cc:2468] T 2e0146fb337d43b2921fb5fd34fb95ed P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 1.
I20260929 02:01:07.820490 16919 leader_election.cc:304] T c6cac2e61f6f4795869af850549096a9 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97df55a0361841319adfae27c9787b2f; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88
I20260929 02:01:07.821025 17253 raft_consensus.cc:2468] T c7892f83a6e34ba0899bb3ac41a749a5 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:07.821722 17623 raft_consensus.cc:3060] T c6cac2e61f6f4795869af850549096a9 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.822713 17595 tablet_bootstrap.cc:654] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.826581 17189 leader_election.cc:304] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc
I20260929 02:01:07.827960 16817 catalog_manager.cc:5838] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88 reported cstate change: term changed from 0 to 1, leader changed from <none> to d103dd4823044faebe46100a0dfbee88 (127.0.71.67). New cstate: current_term: 1 leader_uuid: "d103dd4823044faebe46100a0dfbee88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: HEALTHY } } }
I20260929 02:01:07.828413 17594 tablet_bootstrap.cc:492] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:07.828593 17594 ts_tablet_manager.cc:1449] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.148s user 0.045s sys 0.052s
I20260929 02:01:07.829645 17594 raft_consensus.cc:359] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.829985 17594 raft_consensus.cc:385] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.830109 17594 raft_consensus.cc:740] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.830436 17594 consensus_queue.cc:260] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.831091 17594 ts_tablet_manager.cc:1480] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 02:01:07.831400 17594 tablet_bootstrap.cc:492] T 306f1984e9324a949db3435c56fcfb8f P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:07.833244 17594 tablet_bootstrap.cc:654] T 306f1984e9324a949db3435c56fcfb8f P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.835251 17592 tablet_bootstrap.cc:492] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:07.835421 17592 ts_tablet_manager.cc:1449] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.021s user 0.000s sys 0.006s
I20260929 02:01:07.836221 17592 raft_consensus.cc:359] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.837078 17595 tablet_bootstrap.cc:492] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:07.837251 17595 ts_tablet_manager.cc:1449] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.101s user 0.000s sys 0.003s
I20260929 02:01:07.838102 17595 raft_consensus.cc:359] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.838361 17595 raft_consensus.cc:385] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.838441 17595 raft_consensus.cc:740] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.838732 17595 consensus_queue.cc:260] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.839349 17382 raft_consensus.cc:2804] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:07.839831 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.836413 17592 raft_consensus.cc:385] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.840148 17592 raft_consensus.cc:740] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.840185 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.840487 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.840477 17592 consensus_queue.cc:260] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.840751 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.841540 17592 ts_tablet_manager.cc:1480] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.006s user 0.000s sys 0.003s
I20260929 02:01:07.841933 17592 tablet_bootstrap.cc:492] T c7892f83a6e34ba0899bb3ac41a749a5 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:07.842571 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.842823 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.843070 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.843464 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:07.843725 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:07.844018 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:07.839468 17595 ts_tablet_manager.cc:1480] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 02:01:07.839560 17382 raft_consensus.cc:697] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Becoming Leader. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Running, Role: LEADER
I20260929 02:01:07.844511 17595 tablet_bootstrap.cc:492] T 7239b23c6e30483d9bd0738e07c63610 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:07.844652 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.844877 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.844507 17382 consensus_queue.cc:237] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.847280 17592 tablet_bootstrap.cc:654] T c7892f83a6e34ba0899bb3ac41a749a5 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.849045 17055 leader_election.cc:304] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters: 97df55a0361841319adfae27c9787b2f
I20260929 02:01:07.849579 17369 raft_consensus.cc:2804] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:07.849766 17369 raft_consensus.cc:697] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Becoming Leader. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Running, Role: LEADER
I20260929 02:01:07.850081 17369 consensus_queue.cc:237] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.850353 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.850674 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.850885 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.851085 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.851405 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.851730 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.851868 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.852368 17594 tablet_bootstrap.cc:492] T 306f1984e9324a949db3435c56fcfb8f P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:07.852567 17594 ts_tablet_manager.cc:1449] T 306f1984e9324a949db3435c56fcfb8f P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.005s
I20260929 02:01:07.852560 17623 raft_consensus.cc:2749] T c6cac2e61f6f4795869af850549096a9 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:07.853333 17594 raft_consensus.cc:359] T 306f1984e9324a949db3435c56fcfb8f P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.853482 17595 tablet_bootstrap.cc:654] T 7239b23c6e30483d9bd0738e07c63610 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.853559 17594 raft_consensus.cc:385] T 306f1984e9324a949db3435c56fcfb8f P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.853638 17594 raft_consensus.cc:740] T 306f1984e9324a949db3435c56fcfb8f P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.853883 17594 consensus_queue.cc:260] T 306f1984e9324a949db3435c56fcfb8f P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.854409 17594 ts_tablet_manager.cc:1480] T 306f1984e9324a949db3435c56fcfb8f P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 02:01:07.854725 17594 tablet_bootstrap.cc:492] T 05f61c0da0dd4a20a60cc2b3390cb07c P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:07.857396 17594 tablet_bootstrap.cc:654] T 05f61c0da0dd4a20a60cc2b3390cb07c P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.859052 17592 tablet_bootstrap.cc:492] T c7892f83a6e34ba0899bb3ac41a749a5 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:07.859217 17592 ts_tablet_manager.cc:1449] T c7892f83a6e34ba0899bb3ac41a749a5 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.017s user 0.000s sys 0.003s
I20260929 02:01:07.859956 17592 raft_consensus.cc:359] T c7892f83a6e34ba0899bb3ac41a749a5 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.860174 17592 raft_consensus.cc:385] T c7892f83a6e34ba0899bb3ac41a749a5 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.860260 17592 raft_consensus.cc:740] T c7892f83a6e34ba0899bb3ac41a749a5 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.860543 17592 consensus_queue.cc:260] T c7892f83a6e34ba0899bb3ac41a749a5 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.867441 16818 catalog_manager.cc:5838] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88 reported cstate change: term changed from 0 to 1, leader changed from <none> to d103dd4823044faebe46100a0dfbee88 (127.0.71.67). New cstate: current_term: 1 leader_uuid: "d103dd4823044faebe46100a0dfbee88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: HEALTHY } } }
I20260929 02:01:07.867755 17592 ts_tablet_manager.cc:1480] T c7892f83a6e34ba0899bb3ac41a749a5 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.008s user 0.001s sys 0.002s
I20260929 02:01:07.883162 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.882886 16817 catalog_manager.cc:5838] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97df55a0361841319adfae27c9787b2f (127.0.71.65). New cstate: current_term: 1 leader_uuid: "97df55a0361841319adfae27c9787b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:07.883638 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.884102 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.884588 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.201886
I20260929 02:01:07.885190 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:07.885447 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.888849 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.003s sys 0.002s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":174,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:07.889398 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.889724 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.893834 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.053s user 0.004s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":154,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:07.894584 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.895136 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.895431 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.896860 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.052s user 0.006s sys 0.000s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":786,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":44,"lbm_read_time_us":157,"lbm_reads_lt_1ms":8,"lbm_write_time_us":163,"lbm_writes_lt_1ms":4,"mutex_wait_us":4146,"reinsert_count":0,"update_count":6}
I20260929 02:01:07.897364 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.122276
I20260929 02:01:07.860283 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.009s user 0.002s sys 0.002s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":175,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:07.898075 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.901687 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.012s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":11596}
I20260929 02:01:07.902077 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.012s user 0.004s sys 0.001s Metrics: {"bytes_written":466,"delete_count":0,"lbm_write_time_us":143,"lbm_writes_lt_1ms":4,"mutex_wait_us":6932,"reinsert_count":0,"update_count":2}
I20260929 02:01:07.902513 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":7408}
I20260929 02:01:07.903191 17653 raft_consensus.cc:493] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:07.903407 17653 raft_consensus.cc:515] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.904479 17653 leader_election.cc:290] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:07.905696 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a21bdeb06bb24a45945e00b68b17ca5b" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:07.906206 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a21bdeb06bb24a45945e00b68b17ca5b" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:07.906426 16986 raft_consensus.cc:2468] T a21bdeb06bb24a45945e00b68b17ca5b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:07.906922 17056 leader_election.cc:304] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
W20260929 02:01:07.907475 17055 leader_election.cc:343] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.908002 17653 raft_consensus.cc:2804] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:07.908141 17653 raft_consensus.cc:493] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:07.908255 17653 raft_consensus.cc:3060] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.913758 17653 raft_consensus.cc:515] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.914148 17639 raft_consensus.cc:493] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:07.914319 17639 raft_consensus.cc:515] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.915001 17639 leader_election.cc:290] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:07.915256 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a21bdeb06bb24a45945e00b68b17ca5b" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:07.915462 16986 raft_consensus.cc:3060] T a21bdeb06bb24a45945e00b68b17ca5b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.915678 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b402b4aee2a44330a87657f5a9469dc0" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
W20260929 02:01:07.916254 17186 leader_election.cc:343] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.916920 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b402b4aee2a44330a87657f5a9469dc0" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:07.917132 16983 raft_consensus.cc:2468] T b402b4aee2a44330a87657f5a9469dc0 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 0.
I20260929 02:01:07.917658 17189 leader_election.cc:304] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc
I20260929 02:01:07.918087 17639 raft_consensus.cc:2804] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:07.918210 17639 raft_consensus.cc:493] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:07.918316 17639 raft_consensus.cc:3060] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.918394 16986 raft_consensus.cc:2468] T a21bdeb06bb24a45945e00b68b17ca5b P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:07.919127 17056 leader_election.cc:304] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:07.919457 17398 raft_consensus.cc:2804] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:07.919649 17398 raft_consensus.cc:697] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Becoming Leader. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Running, Role: LEADER
I20260929 02:01:07.920115 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a21bdeb06bb24a45945e00b68b17ca5b" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:07.920046 17398 consensus_queue.cc:237] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
W20260929 02:01:07.920693 17055 leader_election.cc:343] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.920849 17382 raft_consensus.cc:493] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:07.920887 17653 leader_election.cc:290] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:07.920986 17382 raft_consensus.cc:515] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.921614 17382 leader_election.cc:290] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:07.921803 17337 raft_consensus.cc:493] T c7892f83a6e34ba0899bb3ac41a749a5 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:07.921942 17337 raft_consensus.cc:515] T c7892f83a6e34ba0899bb3ac41a749a5 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.922588 17337 leader_election.cc:290] T c7892f83a6e34ba0899bb3ac41a749a5 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:07.923277 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7892f83a6e34ba0899bb3ac41a749a5" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:07.924526 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306f1984e9324a949db3435c56fcfb8f" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:07.924696 17117 raft_consensus.cc:2468] T 306f1984e9324a949db3435c56fcfb8f P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 0.
I20260929 02:01:07.925220 17186 leader_election.cc:304] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:07.925627 17382 raft_consensus.cc:2804] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:07.925751 17382 raft_consensus.cc:493] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:07.925853 17382 raft_consensus.cc:3060] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.929838 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c7892f83a6e34ba0899bb3ac41a749a5" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:07.930101 17253 raft_consensus.cc:2393] T c7892f83a6e34ba0899bb3ac41a749a5 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 97df55a0361841319adfae27c9787b2f in current term 1: Already voted for candidate 8668afa5b5ca4ebdabc812d0789de7bc in this term.
I20260929 02:01:07.930514 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.045s user 0.005s sys 0.006s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5697,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":97,"lbm_read_time_us":450,"lbm_reads_lt_1ms":27,"lbm_write_time_us":337,"lbm_writes_lt_1ms":9,"mutex_wait_us":3717,"peak_mem_usage":32768,"reinsert_count":0,"update_count":2}
I20260929 02:01:07.932315 16921 leader_election.cc:304] T c7892f83a6e34ba0899bb3ac41a749a5 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97df55a0361841319adfae27c9787b2f; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88
I20260929 02:01:07.934067 17639 raft_consensus.cc:515] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.934949 17639 leader_election.cc:290] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:07.935251 17595 tablet_bootstrap.cc:492] T 7239b23c6e30483d9bd0738e07c63610 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:07.935456 17595 ts_tablet_manager.cc:1449] T 7239b23c6e30483d9bd0738e07c63610 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.091s user 0.017s sys 0.040s
I20260929 02:01:07.935772 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.040s user 0.002s sys 0.002s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":152,"lbm_writes_lt_1ms":4,"mutex_wait_us":35149,"reinsert_count":0,"update_count":1}
I20260929 02:01:07.935868 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.040s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":40108}
I20260929 02:01:07.936208 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.038s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":37734}
I20260929 02:01:07.937381 17382 raft_consensus.cc:515] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.937857 17253 raft_consensus.cc:3060] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.938280 17382 leader_election.cc:290] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:07.939263 17595 raft_consensus.cc:359] T 7239b23c6e30483d9bd0738e07c63610 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.939433 17595 raft_consensus.cc:385] T 7239b23c6e30483d9bd0738e07c63610 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.939532 17595 raft_consensus.cc:740] T 7239b23c6e30483d9bd0738e07c63610 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.939775 17595 consensus_queue.cc:260] T 7239b23c6e30483d9bd0738e07c63610 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.940258 17595 ts_tablet_manager.cc:1480] T 7239b23c6e30483d9bd0738e07c63610 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.005s user 0.000s sys 0.000s
I20260929 02:01:07.941102 17595 tablet_bootstrap.cc:492] T bbdc98a938b247bdaf113dc94c3b4d59 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:07.941597 16817 catalog_manager.cc:5838] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66). New cstate: current_term: 1 leader_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:07.943064 17595 tablet_bootstrap.cc:654] T bbdc98a938b247bdaf113dc94c3b4d59 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.937273 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306f1984e9324a949db3435c56fcfb8f" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:07.943971 16986 raft_consensus.cc:2468] T 306f1984e9324a949db3435c56fcfb8f P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 0.
I20260929 02:01:07.937537 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b402b4aee2a44330a87657f5a9469dc0" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:07.944504 16983 raft_consensus.cc:3060] T b402b4aee2a44330a87657f5a9469dc0 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.944597 17120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306f1984e9324a949db3435c56fcfb8f" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:07.944810 17120 raft_consensus.cc:3060] T 306f1984e9324a949db3435c56fcfb8f P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.947563 16983 raft_consensus.cc:2468] T b402b4aee2a44330a87657f5a9469dc0 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 1.
I20260929 02:01:07.947597 17120 raft_consensus.cc:2468] T 306f1984e9324a949db3435c56fcfb8f P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 1.
I20260929 02:01:07.940912 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "306f1984e9324a949db3435c56fcfb8f" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:07.947836 16984 raft_consensus.cc:3060] T 306f1984e9324a949db3435c56fcfb8f P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.949677 17594 tablet_bootstrap.cc:492] T 05f61c0da0dd4a20a60cc2b3390cb07c P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:07.949828 17594 ts_tablet_manager.cc:1449] T 05f61c0da0dd4a20a60cc2b3390cb07c P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.095s user 0.004s sys 0.042s
I20260929 02:01:07.949851 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.052s user 0.010s sys 0.003s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":17780,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":1240,"lbm_read_time_us":519,"lbm_reads_lt_1ms":27,"lbm_write_time_us":428,"lbm_writes_lt_1ms":9,"mutex_wait_us":25898,"peak_mem_usage":32768,"reinsert_count":0,"update_count":8}
I20260929 02:01:07.950408 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.950672 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.950637 17594 raft_consensus.cc:359] T 05f61c0da0dd4a20a60cc2b3390cb07c P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.950837 17594 raft_consensus.cc:385] T 05f61c0da0dd4a20a60cc2b3390cb07c P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.950920 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.950935 17594 raft_consensus.cc:740] T 05f61c0da0dd4a20a60cc2b3390cb07c P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.951148 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.951144 17594 consensus_queue.cc:260] T 05f61c0da0dd4a20a60cc2b3390cb07c P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.951551 16984 raft_consensus.cc:2468] T 306f1984e9324a949db3435c56fcfb8f P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 1.
I20260929 02:01:07.951779 17594 ts_tablet_manager.cc:1480] T 05f61c0da0dd4a20a60cc2b3390cb07c P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 02:01:07.951995 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.952086 17594 tablet_bootstrap.cc:492] T 2e0146fb337d43b2921fb5fd34fb95ed P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:07.952535 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.952953 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.953866 17337 raft_consensus.cc:3060] T c7892f83a6e34ba0899bb3ac41a749a5 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:07.954648 17594 tablet_bootstrap.cc:654] T 2e0146fb337d43b2921fb5fd34fb95ed P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:07.957196 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b402b4aee2a44330a87657f5a9469dc0" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:07.960310 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.960659 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.960934 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.961155 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.962937 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.963172 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.963361 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:07.965637 17624 consensus_queue.cc:1048] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:07.966852 17186 leader_election.cc:304] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters:
W20260929 02:01:07.967284 17186 leader_election.cc:343] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:07.967658 17640 raft_consensus.cc:2804] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:07.967821 17640 raft_consensus.cc:697] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Becoming Leader. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Running, Role: LEADER
I20260929 02:01:07.968029 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.002s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":165,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:07.968115 17640 consensus_queue.cc:237] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.972170 16817 catalog_manager.cc:5838] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66). New cstate: current_term: 1 leader_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:07.973294 17189 leader_election.cc:304] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc
I20260929 02:01:07.975558 17398 consensus_queue.cc:1048] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:07.977097 17623 consensus_queue.cc:1048] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:07.977691 17623 consensus_queue.cc:1048] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:07.981805 17640 raft_consensus.cc:2804] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:07.982009 17640 raft_consensus.cc:697] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Becoming Leader. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Running, Role: LEADER
I20260929 02:01:07.982220 17623 consensus_queue.cc:1048] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:07.982358 17640 consensus_queue.cc:237] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.985780 17594 tablet_bootstrap.cc:492] T 2e0146fb337d43b2921fb5fd34fb95ed P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:07.985972 17594 ts_tablet_manager.cc:1449] T 2e0146fb337d43b2921fb5fd34fb95ed P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.034s user 0.006s sys 0.003s
I20260929 02:01:07.987982 16818 catalog_manager.cc:5838] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88 reported cstate change: term changed from 0 to 1, leader changed from <none> to d103dd4823044faebe46100a0dfbee88 (127.0.71.67). New cstate: current_term: 1 leader_uuid: "d103dd4823044faebe46100a0dfbee88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: HEALTHY } } }
I20260929 02:01:07.988232 17398 consensus_queue.cc:1048] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:07.988941 17594 raft_consensus.cc:359] T 2e0146fb337d43b2921fb5fd34fb95ed P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.989759 17594 raft_consensus.cc:385] T 2e0146fb337d43b2921fb5fd34fb95ed P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:07.989863 17594 raft_consensus.cc:740] T 2e0146fb337d43b2921fb5fd34fb95ed P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:07.990402 17594 consensus_queue.cc:260] T 2e0146fb337d43b2921fb5fd34fb95ed P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:07.990592 17337 raft_consensus.cc:2749] T c7892f83a6e34ba0899bb3ac41a749a5 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:07.991487 17594 ts_tablet_manager.cc:1480] T 2e0146fb337d43b2921fb5fd34fb95ed P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260929 02:01:07.991696 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.991838 17594 tablet_bootstrap.cc:492] T b402b4aee2a44330a87657f5a9469dc0 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:07.968641 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.145019
I20260929 02:01:07.992028 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.992249 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.992599 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.992817 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:07.997829 17594 tablet_bootstrap.cc:654] T b402b4aee2a44330a87657f5a9469dc0 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:08.008425 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.016s user 0.004s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":181,"lbm_writes_lt_1ms":4,"mutex_wait_us":8524,"reinsert_count":0,"update_count":1}
I20260929 02:01:08.014838 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.022s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":21738}
I20260929 02:01:08.015218 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.023s user 0.004s sys 0.002s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":166,"lbm_writes_lt_1ms":4,"mutex_wait_us":15772,"reinsert_count":0,"update_count":1}
I20260929 02:01:08.016076 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.003000
I20260929 02:01:08.016393 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.003000
I20260929 02:01:08.016798 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.040728 17674 consensus_queue.cc:1048] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.006s
I20260929 02:01:08.044404 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.049245 17337 consensus_queue.cc:1048] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:08.049355 17382 consensus_queue.cc:1048] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.058s
I20260929 02:01:08.058027 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.061s user 0.004s sys 0.000s Metrics: {"bytes_written":466,"delete_count":0,"lbm_write_time_us":154,"lbm_writes_lt_1ms":4,"mutex_wait_us":55879,"reinsert_count":0,"update_count":2}
I20260929 02:01:08.059324 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.066s user 0.013s sys 0.001s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5771,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":17993,"dirs.run_cpu_time_us":396,"dirs.run_wall_time_us":2196,"lbm_read_time_us":737,"lbm_reads_lt_1ms":23,"lbm_write_time_us":779,"lbm_writes_lt_1ms":9,"mutex_wait_us":17849,"peak_mem_usage":32768,"reinsert_count":0,"update_count":2}
I20260929 02:01:08.060886 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.062s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":62172}
I20260929 02:01:08.060963 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.042s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":42041}
I20260929 02:01:08.061952 17337 raft_consensus.cc:493] T 5acedab1b27c414fa5181e35aa9ac24b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:08.062127 17337 raft_consensus.cc:515] T 5acedab1b27c414fa5181e35aa9ac24b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
W20260929 02:01:08.062325 17189 outbound_call.cc:321] RPC callback for RPC call kudu.consensus.ConsensusService.UpdateConsensus -> {remote=127.0.71.65:43625, user_credentials={real_user=slave}} blocked reactor thread for 87287us
I20260929 02:01:08.062947 17673 raft_consensus.cc:493] T 7239b23c6e30483d9bd0738e07c63610 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:08.063226 17673 raft_consensus.cc:515] T 7239b23c6e30483d9bd0738e07c63610 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
W20260929 02:01:08.064292 17186 consensus_peers.cc:597] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88 -> Peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Couldn't send request to peer 8668afa5b5ca4ebdabc812d0789de7bc. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:01:08.064806 17337 leader_election.cc:290] T 5acedab1b27c414fa5181e35aa9ac24b P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:08.064949 17594 tablet_bootstrap.cc:492] T b402b4aee2a44330a87657f5a9469dc0 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:08.065126 17594 ts_tablet_manager.cc:1449] T b402b4aee2a44330a87657f5a9469dc0 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.073s user 0.012s sys 0.012s
I20260929 02:01:08.065201 17116 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7239b23c6e30483d9bd0738e07c63610" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:08.065578 17116 raft_consensus.cc:2393] T 7239b23c6e30483d9bd0738e07c63610 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d103dd4823044faebe46100a0dfbee88 in current term 1: Already voted for candidate 97df55a0361841319adfae27c9787b2f in this term.
I20260929 02:01:08.066095 17594 raft_consensus.cc:359] T b402b4aee2a44330a87657f5a9469dc0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.065997 17115 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5acedab1b27c414fa5181e35aa9ac24b" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:08.066320 17594 raft_consensus.cc:385] T b402b4aee2a44330a87657f5a9469dc0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:08.066422 17594 raft_consensus.cc:740] T b402b4aee2a44330a87657f5a9469dc0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:08.066195 17673 leader_election.cc:290] T 7239b23c6e30483d9bd0738e07c63610 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:08.066673 17594 consensus_queue.cc:260] T b402b4aee2a44330a87657f5a9469dc0 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.067195 17594 ts_tablet_manager.cc:1480] T b402b4aee2a44330a87657f5a9469dc0 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:01:08.067466 17594 tablet_bootstrap.cc:492] T 8f328eef30d541c7a61e7d7403e597c7 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:08.069664 17594 tablet_bootstrap.cc:654] T 8f328eef30d541c7a61e7d7403e597c7 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:08.003130 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.051s user 0.007s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":164,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:08.067519 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "7239b23c6e30483d9bd0738e07c63610" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:08.073356 16984 raft_consensus.cc:3060] T 5acedab1b27c414fa5181e35aa9ac24b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:08.081979 17189 leader_election.cc:304] T 7239b23c6e30483d9bd0738e07c63610 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f
I20260929 02:01:08.082751 17673 raft_consensus.cc:3060] T 7239b23c6e30483d9bd0738e07c63610 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:08.086948 17673 raft_consensus.cc:2749] T 7239b23c6e30483d9bd0738e07c63610 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:08.095870 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.079s user 0.003s sys 0.006s Metrics: {"cfile_cache_miss":6,"cfile_cache_miss_bytes":1005,"cfile_init":2,"delete_count":0,"delta_blocks_compacted":3,"delta_iterators_relevant":3,"dirs.queue_time_us":68531,"lbm_read_time_us":311,"lbm_reads_lt_1ms":14,"lbm_write_time_us":186,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":3}
I20260929 02:01:08.108191 17673 consensus_queue.cc:1048] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:08.150120 17369 raft_consensus.cc:493] T 05f61c0da0dd4a20a60cc2b3390cb07c P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:08.150390 17369 raft_consensus.cc:515] T 05f61c0da0dd4a20a60cc2b3390cb07c P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.151818 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05f61c0da0dd4a20a60cc2b3390cb07c" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:08.152145 16986 raft_consensus.cc:2393] T 05f61c0da0dd4a20a60cc2b3390cb07c P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8668afa5b5ca4ebdabc812d0789de7bc in current term 1: Already voted for candidate d103dd4823044faebe46100a0dfbee88 in this term.
I20260929 02:01:08.153908 17369 leader_election.cc:290] T 05f61c0da0dd4a20a60cc2b3390cb07c P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:08.154034 17252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "05f61c0da0dd4a20a60cc2b3390cb07c" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:08.156642 17055 leader_election.cc:304] T 05f61c0da0dd4a20a60cc2b3390cb07c P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc; no voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88
I20260929 02:01:08.157176 17369 raft_consensus.cc:3060] T 05f61c0da0dd4a20a60cc2b3390cb07c P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:08.167906 17369 raft_consensus.cc:2749] T 05f61c0da0dd4a20a60cc2b3390cb07c P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:08.177153 17337 raft_consensus.cc:493] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:08.177371 17337 raft_consensus.cc:515] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.178412 17337 leader_election.cc:290] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:08.182566 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbdc98a938b247bdaf113dc94c3b4d59" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:08.182863 17118 raft_consensus.cc:2468] T bbdc98a938b247bdaf113dc94c3b4d59 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:08.183467 16919 leader_election.cc:304] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:08.183879 17337 raft_consensus.cc:2804] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:08.183995 17337 raft_consensus.cc:493] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:08.184119 17337 raft_consensus.cc:3060] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:08.190975 17252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5acedab1b27c414fa5181e35aa9ac24b" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:08.191044 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbdc98a938b247bdaf113dc94c3b4d59" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:08.191774 16921 leader_election.cc:304] T 5acedab1b27c414fa5181e35aa9ac24b P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97df55a0361841319adfae27c9787b2f; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88
W20260929 02:01:08.192562 16921 leader_election.cc:343] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:08.193261 17624 consensus_queue.cc:1048] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:08.203217 17625 raft_consensus.cc:2749] T 5acedab1b27c414fa5181e35aa9ac24b P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:08.206931 16818 catalog_manager.cc:5838] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88 reported cstate change: term changed from 0 to 1, leader changed from <none> to d103dd4823044faebe46100a0dfbee88 (127.0.71.67). New cstate: current_term: 1 leader_uuid: "d103dd4823044faebe46100a0dfbee88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: HEALTHY } } }
I20260929 02:01:08.198709 17337 raft_consensus.cc:515] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.209829 17250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbdc98a938b247bdaf113dc94c3b4d59" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:08.210182 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbdc98a938b247bdaf113dc94c3b4d59" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:08.210418 17118 raft_consensus.cc:3060] T bbdc98a938b247bdaf113dc94c3b4d59 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:08.210876 17595 tablet_bootstrap.cc:492] T bbdc98a938b247bdaf113dc94c3b4d59 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:08.211040 17595 ts_tablet_manager.cc:1449] T bbdc98a938b247bdaf113dc94c3b4d59 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.270s user 0.056s sys 0.085s
I20260929 02:01:08.211803 17595 raft_consensus.cc:359] T bbdc98a938b247bdaf113dc94c3b4d59 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.211992 17595 raft_consensus.cc:385] T bbdc98a938b247bdaf113dc94c3b4d59 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:08.212082 17595 raft_consensus.cc:740] T bbdc98a938b247bdaf113dc94c3b4d59 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:08.212314 17595 consensus_queue.cc:260] T bbdc98a938b247bdaf113dc94c3b4d59 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
W20260929 02:01:08.212677 16921 leader_election.cc:343] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:08.214322 17118 raft_consensus.cc:2468] T bbdc98a938b247bdaf113dc94c3b4d59 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:08.214617 17595 ts_tablet_manager.cc:1480] T bbdc98a938b247bdaf113dc94c3b4d59 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 02:01:08.214942 17595 tablet_bootstrap.cc:492] T a21bdeb06bb24a45945e00b68b17ca5b P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:08.214980 16919 leader_election.cc:304] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters: d103dd4823044faebe46100a0dfbee88
I20260929 02:01:08.215378 17625 raft_consensus.cc:2804] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:08.215509 17625 raft_consensus.cc:697] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Becoming Leader. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Running, Role: LEADER
I20260929 02:01:08.215803 17625 consensus_queue.cc:237] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.216887 17595 tablet_bootstrap.cc:654] T a21bdeb06bb24a45945e00b68b17ca5b P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:08.209216 17337 leader_election.cc:290] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Requested vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:08.226263 17594 tablet_bootstrap.cc:492] T 8f328eef30d541c7a61e7d7403e597c7 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:08.226464 17594 ts_tablet_manager.cc:1449] T 8f328eef30d541c7a61e7d7403e597c7 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.159s user 0.050s sys 0.087s
I20260929 02:01:08.227429 17594 raft_consensus.cc:359] T 8f328eef30d541c7a61e7d7403e597c7 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.227661 17594 raft_consensus.cc:385] T 8f328eef30d541c7a61e7d7403e597c7 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:08.227749 17594 raft_consensus.cc:740] T 8f328eef30d541c7a61e7d7403e597c7 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:08.228500 17595 tablet_bootstrap.cc:492] T a21bdeb06bb24a45945e00b68b17ca5b P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:08.228472 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.228699 17595 ts_tablet_manager.cc:1449] T a21bdeb06bb24a45945e00b68b17ca5b P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.014s user 0.006s sys 0.004s
I20260929 02:01:08.228817 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.228411 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.229123 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.229353 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.229555 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.229599 17595 raft_consensus.cc:359] T a21bdeb06bb24a45945e00b68b17ca5b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.229785 17595 raft_consensus.cc:385] T a21bdeb06bb24a45945e00b68b17ca5b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:08.229888 17595 raft_consensus.cc:740] T a21bdeb06bb24a45945e00b68b17ca5b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:08.230110 17595 consensus_queue.cc:260] T a21bdeb06bb24a45945e00b68b17ca5b P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.230702 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.230995 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.231504 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.231840 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.233037 17595 ts_tablet_manager.cc:1480] T a21bdeb06bb24a45945e00b68b17ca5b P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.004s user 0.000s sys 0.001s
I20260929 02:01:08.233759 16817 catalog_manager.cc:5838] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97df55a0361841319adfae27c9787b2f (127.0.71.65). New cstate: current_term: 1 leader_uuid: "97df55a0361841319adfae27c9787b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:08.235205 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.201906
I20260929 02:01:08.235462 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.201906
I20260929 02:01:08.235703 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.201906
I20260929 02:01:08.235810 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.002s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":162,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:08.236011 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.236229 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.236307 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:08.236377 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:08.236646 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:08.236925 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:08.237257 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.237545 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.228070 17594 consensus_queue.cc:260] T 8f328eef30d541c7a61e7d7403e597c7 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.239885 17594 ts_tablet_manager.cc:1480] T 8f328eef30d541c7a61e7d7403e597c7 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.013s user 0.001s sys 0.001s
I20260929 02:01:08.241125 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.245980 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.009s user 0.001s sys 0.005s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":721,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":270,"lbm_read_time_us":147,"lbm_reads_lt_1ms":8,"lbm_write_time_us":172,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":4}
I20260929 02:01:08.246523 17118 raft_consensus.cc:3060] T 2e0146fb337d43b2921fb5fd34fb95ed P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:08.249773 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.250130 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.250458 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.250763 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.251116 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.251117 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"spinlock_wait_cycles":896}
I20260929 02:01:08.251585 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:08.251863 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:08.251873 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.252187 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.250787 17680 consensus_queue.cc:1048] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.002s
I20260929 02:01:08.252673 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.256011 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.005s user 0.001s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":124,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:08.262339 17680 raft_consensus.cc:493] T a21bdeb06bb24a45945e00b68b17ca5b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:08.263736 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.034s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":172,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:08.263988 17680 raft_consensus.cc:515] T a21bdeb06bb24a45945e00b68b17ca5b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.264544 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.264657 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.013s user 0.005s sys 0.002s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":740,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":54,"lbm_read_time_us":175,"lbm_reads_lt_1ms":8,"lbm_write_time_us":163,"lbm_writes_lt_1ms":4,"mutex_wait_us":3884,"reinsert_count":0,"update_count":5}
I20260929 02:01:08.265358 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a21bdeb06bb24a45945e00b68b17ca5b" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:08.265419 17680 leader_election.cc:290] T a21bdeb06bb24a45945e00b68b17ca5b P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:08.265481 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.265686 16984 raft_consensus.cc:2393] T a21bdeb06bb24a45945e00b68b17ca5b P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d103dd4823044faebe46100a0dfbee88 in current term 1: Already voted for candidate 8668afa5b5ca4ebdabc812d0789de7bc in this term.
I20260929 02:01:08.266214 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a21bdeb06bb24a45945e00b68b17ca5b" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:08.267529 17189 leader_election.cc:304] T a21bdeb06bb24a45945e00b68b17ca5b P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f
I20260929 02:01:08.267835 17680 raft_consensus.cc:3060] T a21bdeb06bb24a45945e00b68b17ca5b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:08.268896 17625 consensus_queue.cc:1048] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:08.271934 17680 raft_consensus.cc:2749] T a21bdeb06bb24a45945e00b68b17ca5b P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:08.273440 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.038s user 0.011s sys 0.001s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5875,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":84,"lbm_read_time_us":483,"lbm_reads_lt_1ms":27,"lbm_write_time_us":349,"lbm_writes_lt_1ms":9,"mutex_wait_us":21150,"peak_mem_usage":32768,"reinsert_count":0,"update_count":3}
I20260929 02:01:08.279376 17398 consensus_queue.cc:1048] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:08.280248 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.028s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":27611}
I20260929 02:01:08.280881 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.016s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":15500}
I20260929 02:01:08.281356 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.029s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":163,"lbm_writes_lt_1ms":4,"mutex_wait_us":21168,"reinsert_count":0,"update_count":1}
I20260929 02:01:08.283888 17680 consensus_queue.cc:1048] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:08.288698 17653 consensus_queue.cc:1048] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:08.294708 17623 consensus_queue.cc:1048] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:08.322410 17398 raft_consensus.cc:493] T b402b4aee2a44330a87657f5a9469dc0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:08.322602 17398 raft_consensus.cc:515] T b402b4aee2a44330a87657f5a9469dc0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.323407 17398 leader_election.cc:290] T b402b4aee2a44330a87657f5a9469dc0 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:08.323992 17250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b402b4aee2a44330a87657f5a9469dc0" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:08.323993 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b402b4aee2a44330a87657f5a9469dc0" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:08.324297 16983 raft_consensus.cc:2393] T b402b4aee2a44330a87657f5a9469dc0 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8668afa5b5ca4ebdabc812d0789de7bc in current term 1: Already voted for candidate d103dd4823044faebe46100a0dfbee88 in this term.
I20260929 02:01:08.324808 17056 leader_election.cc:304] T b402b4aee2a44330a87657f5a9469dc0 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc; no voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88
I20260929 02:01:08.325122 17398 raft_consensus.cc:3060] T b402b4aee2a44330a87657f5a9469dc0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:08.327405 17398 raft_consensus.cc:2749] T b402b4aee2a44330a87657f5a9469dc0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:08.348052 17680 consensus_queue.cc:1048] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:08.352630 17674 consensus_queue.cc:1048] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:08.367794 17674 consensus_queue.cc:1048] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:08.372179 17682 consensus_queue.cc:1048] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:08.381618 17682 raft_consensus.cc:493] T bbdc98a938b247bdaf113dc94c3b4d59 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:08.381904 17682 raft_consensus.cc:515] T bbdc98a938b247bdaf113dc94c3b4d59 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.383248 17682 leader_election.cc:290] T bbdc98a938b247bdaf113dc94c3b4d59 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:08.383617 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbdc98a938b247bdaf113dc94c3b4d59" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:08.383752 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "bbdc98a938b247bdaf113dc94c3b4d59" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:08.384090 17118 raft_consensus.cc:2393] T bbdc98a938b247bdaf113dc94c3b4d59 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d103dd4823044faebe46100a0dfbee88 in current term 1: Already voted for candidate 97df55a0361841319adfae27c9787b2f in this term.
I20260929 02:01:08.384568 17186 leader_election.cc:304] T bbdc98a938b247bdaf113dc94c3b4d59 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f
I20260929 02:01:08.384881 17682 raft_consensus.cc:3060] T bbdc98a938b247bdaf113dc94c3b4d59 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:08.386803 17682 raft_consensus.cc:2749] T bbdc98a938b247bdaf113dc94c3b4d59 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:08.406886 17682 consensus_queue.cc:1048] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:08.412796 17680 consensus_queue.cc:1048] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:08.443449 17398 consensus_queue.cc:1048] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:08.445616 16818 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44708:
table { table_name: "table-2" } modify_external_catalogs: true
I20260929 02:01:08.445916 16818 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44708:
table { table_name: "table-2" } modify_external_catalogs: true
I20260929 02:01:08.449353 17543 consensus_queue.cc:1048] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:08.464479 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.464843 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.465170 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.465446 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.465808 17218 tablet_service.cc:1558] Processing DeleteTablet for tablet a21bdeb06bb24a45945e00b68b17ca5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:08.466495 17706 tablet_replica.cc:333] T a21bdeb06bb24a45945e00b68b17ca5b P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:08.466655 17706 raft_consensus.cc:2243] T a21bdeb06bb24a45945e00b68b17ca5b P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:08.466862 17706 raft_consensus.cc:2272] T a21bdeb06bb24a45945e00b68b17ca5b P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.467360 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.467669 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.467785 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.427156
I20260929 02:01:08.467901 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.467988 17706 ts_tablet_manager.cc:1962] T a21bdeb06bb24a45945e00b68b17ca5b P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.468060 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.427156
I20260929 02:01:08.468304 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.468940 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.469239 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.469488 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.465654 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet d9511e2734f1414ba6df2e7c9459ce34
I20260929 02:01:08.470404 17218 tablet_service.cc:1558] Processing DeleteTablet for tablet d9511e2734f1414ba6df2e7c9459ce34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:08.470238 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 925a0eec8efb496ba538b7f7c44d84f2
I20260929 02:01:08.471239 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.471367 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 5b2b8f6f05b74c849d19705a2e8ab0e7
I20260929 02:01:08.471412 17218 tablet_service.cc:1558] Processing DeleteTablet for tablet 925a0eec8efb496ba538b7f7c44d84f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:08.471530 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.472121 17098 tablet_service.cc:1558] Processing DeleteTablet for tablet a21bdeb06bb24a45945e00b68b17ca5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:08.472317 17085 tablet_service.cc:1558] Processing DeleteTablet for tablet d9511e2734f1414ba6df2e7c9459ce34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:08.472538 17218 tablet_service.cc:1558] Processing DeleteTablet for tablet 5b2b8f6f05b74c849d19705a2e8ab0e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:08.472563 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 167d937681dc4e509f1d5d4ce2358d27
I20260929 02:01:08.472564 17095 tablet_service.cc:1558] Processing DeleteTablet for tablet 5b2b8f6f05b74c849d19705a2e8ab0e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:08.472755 17092 tablet_service.cc:1558] Processing DeleteTablet for tablet 925a0eec8efb496ba538b7f7c44d84f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:08.473440 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 5acedab1b27c414fa5181e35aa9ac24b
I20260929 02:01:08.473465 17218 tablet_service.cc:1558] Processing DeleteTablet for tablet 167d937681dc4e509f1d5d4ce2358d27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:08.474272 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet c6cac2e61f6f4795869af850549096a9
I20260929 02:01:08.474241 17095 tablet_service.cc:1558] Processing DeleteTablet for tablet 5acedab1b27c414fa5181e35aa9ac24b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:08.474298 17218 tablet_service.cc:1558] Processing DeleteTablet for tablet 5acedab1b27c414fa5181e35aa9ac24b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:08.474485 17092 tablet_service.cc:1558] Processing DeleteTablet for tablet 167d937681dc4e509f1d5d4ce2358d27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:08.475040 17706 ts_tablet_manager.cc:1975] T a21bdeb06bb24a45945e00b68b17ca5b P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.475118 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet ae7f6651c994403697a0f04508b70dae
I20260929 02:01:08.475131 17218 tablet_service.cc:1558] Processing DeleteTablet for tablet c6cac2e61f6f4795869af850549096a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:08.475198 17706 log.cc:1199] T a21bdeb06bb24a45945e00b68b17ca5b P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/a21bdeb06bb24a45945e00b68b17ca5b
I20260929 02:01:08.475744 17706 ts_tablet_manager.cc:1996] T a21bdeb06bb24a45945e00b68b17ca5b P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:08.476028 17218 tablet_service.cc:1558] Processing DeleteTablet for tablet ae7f6651c994403697a0f04508b70dae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:08.476431 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet c7892f83a6e34ba0899bb3ac41a749a5
I20260929 02:01:08.477079 17706 tablet_replica.cc:333] T d9511e2734f1414ba6df2e7c9459ce34 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:08.477383 17706 raft_consensus.cc:2243] T d9511e2734f1414ba6df2e7c9459ce34 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:08.477420 17085 tablet_service.cc:1558] Processing DeleteTablet for tablet c7892f83a6e34ba0899bb3ac41a749a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:08.477433 17218 tablet_service.cc:1558] Processing DeleteTablet for tablet c7892f83a6e34ba0899bb3ac41a749a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:08.477640 17706 raft_consensus.cc:2272] T d9511e2734f1414ba6df2e7c9459ce34 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.477756 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 306f1984e9324a949db3435c56fcfb8f
I20260929 02:01:08.478506 17218 tablet_service.cc:1558] Processing DeleteTablet for tablet 306f1984e9324a949db3435c56fcfb8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:08.478514 17085 tablet_service.cc:1558] Processing DeleteTablet for tablet 306f1984e9324a949db3435c56fcfb8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:08.478717 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet bbdc98a938b247bdaf113dc94c3b4d59
I20260929 02:01:08.478720 17706 ts_tablet_manager.cc:1962] T d9511e2734f1414ba6df2e7c9459ce34 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.479545 17218 tablet_service.cc:1558] Processing DeleteTablet for tablet bbdc98a938b247bdaf113dc94c3b4d59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:08.479573 17085 tablet_service.cc:1558] Processing DeleteTablet for tablet bbdc98a938b247bdaf113dc94c3b4d59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:08.479765 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 7239b23c6e30483d9bd0738e07c63610
I20260929 02:01:08.480616 17218 tablet_service.cc:1558] Processing DeleteTablet for tablet 7239b23c6e30483d9bd0738e07c63610 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:08.480629 17085 tablet_service.cc:1558] Processing DeleteTablet for tablet 7239b23c6e30483d9bd0738e07c63610 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:08.480674 17092 tablet_service.cc:1558] Processing DeleteTablet for tablet c6cac2e61f6f4795869af850549096a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:08.480831 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 8f328eef30d541c7a61e7d7403e597c7
I20260929 02:01:08.480845 17095 tablet_service.cc:1558] Processing DeleteTablet for tablet ae7f6651c994403697a0f04508b70dae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:08.481600 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet a21bdeb06bb24a45945e00b68b17ca5b
I20260929 02:01:08.481626 17218 tablet_service.cc:1558] Processing DeleteTablet for tablet 8f328eef30d541c7a61e7d7403e597c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:08.482247 17095 tablet_service.cc:1558] Processing DeleteTablet for tablet 8f328eef30d541c7a61e7d7403e597c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:08.482497 17092 tablet_service.cc:1558] Processing DeleteTablet for tablet a21bdeb06bb24a45945e00b68b17ca5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:08.482831 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 2e0146fb337d43b2921fb5fd34fb95ed
I20260929 02:01:08.482867 17218 tablet_service.cc:1558] Processing DeleteTablet for tablet a21bdeb06bb24a45945e00b68b17ca5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:08.484025 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet b402b4aee2a44330a87657f5a9469dc0
I20260929 02:01:08.484062 17218 tablet_service.cc:1558] Processing DeleteTablet for tablet 2e0146fb337d43b2921fb5fd34fb95ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:08.485777 17095 tablet_service.cc:1558] Processing DeleteTablet for tablet b402b4aee2a44330a87657f5a9469dc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:08.486088 17092 tablet_service.cc:1558] Processing DeleteTablet for tablet 2e0146fb337d43b2921fb5fd34fb95ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:08.486244 16818 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 05f61c0da0dd4a20a60cc2b3390cb07c
I20260929 02:01:08.486279 17218 tablet_service.cc:1558] Processing DeleteTablet for tablet b402b4aee2a44330a87657f5a9469dc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:08.486917 17092 tablet_service.cc:1558] Processing DeleteTablet for tablet 05f61c0da0dd4a20a60cc2b3390cb07c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:08.487351 17706 ts_tablet_manager.cc:1975] T d9511e2734f1414ba6df2e7c9459ce34 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.487399 17218 tablet_service.cc:1558] Processing DeleteTablet for tablet 05f61c0da0dd4a20a60cc2b3390cb07c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:08.487574 17706 log.cc:1199] T d9511e2734f1414ba6df2e7c9459ce34 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/d9511e2734f1414ba6df2e7c9459ce34
I20260929 02:01:08.488034 17706 ts_tablet_manager.cc:1996] T d9511e2734f1414ba6df2e7c9459ce34 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:08.488057 17707 tablet_replica.cc:333] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:08.488250 17707 raft_consensus.cc:2243] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:08.488911 16962 tablet_service.cc:1558] Processing DeleteTablet for tablet a21bdeb06bb24a45945e00b68b17ca5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:08.488898 16818 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44708:
name: "table-3"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "col"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 16
seed: 0
}
}
I20260929 02:01:08.489481 17707 raft_consensus.cc:2272] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.489602 16956 tablet_service.cc:1558] Processing DeleteTablet for tablet 925a0eec8efb496ba538b7f7c44d84f2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:08.489629 16953 tablet_service.cc:1558] Processing DeleteTablet for tablet 5b2b8f6f05b74c849d19705a2e8ab0e7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:08.489854 16961 tablet_service.cc:1558] Processing DeleteTablet for tablet 167d937681dc4e509f1d5d4ce2358d27 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:08.489990 16955 tablet_service.cc:1558] Processing DeleteTablet for tablet d9511e2734f1414ba6df2e7c9459ce34 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:08.490041 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.021s user 0.005s sys 0.000s Metrics: {"bytes_written":905,"delete_count":0,"lbm_write_time_us":180,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":17}
W20260929 02:01:08.490253 16818 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table-3 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:01:08.490530 17707 ts_tablet_manager.cc:1962] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.495965 17706 tablet_replica.cc:333] T 925a0eec8efb496ba538b7f7c44d84f2 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:08.496025 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet d9511e2734f1414ba6df2e7c9459ce34 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:08.496738 17706 raft_consensus.cc:2243] T 925a0eec8efb496ba538b7f7c44d84f2 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:08.496968 17706 raft_consensus.cc:2272] T 925a0eec8efb496ba538b7f7c44d84f2 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.497421 17707 ts_tablet_manager.cc:1975] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.497553 17707 log.cc:1199] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/a21bdeb06bb24a45945e00b68b17ca5b
I20260929 02:01:08.498035 17707 ts_tablet_manager.cc:1996] T a21bdeb06bb24a45945e00b68b17ca5b P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:08.498330 17706 ts_tablet_manager.cc:1962] T 925a0eec8efb496ba538b7f7c44d84f2 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.499061 17707 tablet_replica.cc:333] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:08.499236 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet a21bdeb06bb24a45945e00b68b17ca5b (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:08.499387 17707 raft_consensus.cc:2243] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:08.499567 16955 tablet_service.cc:1558] Processing DeleteTablet for tablet 5acedab1b27c414fa5181e35aa9ac24b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:08.499728 16961 tablet_service.cc:1558] Processing DeleteTablet for tablet c6cac2e61f6f4795869af850549096a9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:08.499770 17707 raft_consensus.cc:2272] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.499925 16953 tablet_service.cc:1558] Processing DeleteTablet for tablet ae7f6651c994403697a0f04508b70dae with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:08.500471 16953 tablet_service.cc:1558] Processing DeleteTablet for tablet c7892f83a6e34ba0899bb3ac41a749a5 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:08.501057 16961 tablet_service.cc:1558] Processing DeleteTablet for tablet 306f1984e9324a949db3435c56fcfb8f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:08.501940 16953 tablet_service.cc:1558] Processing DeleteTablet for tablet bbdc98a938b247bdaf113dc94c3b4d59 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:08.502485 16953 tablet_service.cc:1558] Processing DeleteTablet for tablet 7239b23c6e30483d9bd0738e07c63610 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:08.502389 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.502859 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.216360
I20260929 02:01:08.502935 16953 tablet_service.cc:1558] Processing DeleteTablet for tablet 8f328eef30d541c7a61e7d7403e597c7 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:08.503196 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.216360
I20260929 02:01:08.503391 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.038s user 0.003s sys 0.003s Metrics: {"bytes_written":905,"delete_count":0,"lbm_write_time_us":164,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":17}
I20260929 02:01:08.503823 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.216360
I20260929 02:01:08.507567 17706 ts_tablet_manager.cc:1975] T 925a0eec8efb496ba538b7f7c44d84f2 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.507668 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.507942 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.508530 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.511766 16953 tablet_service.cc:1558] Processing DeleteTablet for tablet a21bdeb06bb24a45945e00b68b17ca5b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:08.512468 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:08.512738 16953 tablet_service.cc:1558] Processing DeleteTablet for tablet 2e0146fb337d43b2921fb5fd34fb95ed with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:08.513159 16953 tablet_service.cc:1558] Processing DeleteTablet for tablet b402b4aee2a44330a87657f5a9469dc0 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:08.513607 16953 tablet_service.cc:1558] Processing DeleteTablet for tablet 05f61c0da0dd4a20a60cc2b3390cb07c with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:08 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:08.514029 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.520251 17706 log.cc:1199] T 925a0eec8efb496ba538b7f7c44d84f2 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/925a0eec8efb496ba538b7f7c44d84f2
I20260929 02:01:08.521052 17706 ts_tablet_manager.cc:1996] T 925a0eec8efb496ba538b7f7c44d84f2 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:08.521111 17707 ts_tablet_manager.cc:1962] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.522682 17706 tablet_replica.cc:333] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:08.522841 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 925a0eec8efb496ba538b7f7c44d84f2 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:08.524102 17706 raft_consensus.cc:2243] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:08.524641 17706 raft_consensus.cc:2272] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.526135 17706 ts_tablet_manager.cc:1962] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.526971 17218 tablet_service.cc:1511] Processing CreateTablet for tablet 5e9b68ceb78f45edaaebaf7314364446 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.527403 17218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e9b68ceb78f45edaaebaf7314364446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.531589 17707 ts_tablet_manager.cc:1975] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.532119 17707 log.cc:1199] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/d9511e2734f1414ba6df2e7c9459ce34
I20260929 02:01:08.532855 17707 ts_tablet_manager.cc:1996] T d9511e2734f1414ba6df2e7c9459ce34 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:08.534072 17707 tablet_replica.cc:333] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:08.534297 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet d9511e2734f1414ba6df2e7c9459ce34 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:08.535413 17707 raft_consensus.cc:2243] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:08.535705 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.068s user 0.014s sys 0.000s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":18485,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":7167,"dirs.run_cpu_time_us":258,"dirs.run_wall_time_us":1255,"lbm_read_time_us":558,"lbm_reads_lt_1ms":27,"lbm_write_time_us":390,"lbm_writes_lt_1ms":9,"mutex_wait_us":10721,"peak_mem_usage":32768,"reinsert_count":0,"thread_start_us":3650,"threads_started":1,"update_count":22}
I20260929 02:01:08.536602 17707 raft_consensus.cc:2272] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.537299 17708 tablet_replica.cc:333] T a21bdeb06bb24a45945e00b68b17ca5b P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:08.537591 17708 raft_consensus.cc:2243] T a21bdeb06bb24a45945e00b68b17ca5b P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:08.537676 17707 ts_tablet_manager.cc:1962] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.537961 17708 raft_consensus.cc:2272] T a21bdeb06bb24a45945e00b68b17ca5b P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.538105 17595 tablet_bootstrap.cc:492] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:08.540122 17595 tablet_bootstrap.cc:654] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:08.541676 17708 ts_tablet_manager.cc:1962] T a21bdeb06bb24a45945e00b68b17ca5b P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.543195 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.033s user 0.015s sys 0.000s Metrics: {"bytes_written":711,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":76,"dirs.run_cpu_time_us":177,"dirs.run_wall_time_us":1531,"drs_written":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4,"lbm_write_time_us":1506,"lbm_writes_lt_1ms":31,"peak_mem_usage":0,"reinsert_count":0,"rows_written":17,"update_count":16}
I20260929 02:01:08.544219 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.356749
I20260929 02:01:08.544529 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260929 02:01:08.544780 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260929 02:01:08.545094 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260929 02:01:08.545176 17707 ts_tablet_manager.cc:1975] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.545343 17707 log.cc:1199] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/5b2b8f6f05b74c849d19705a2e8ab0e7
I20260929 02:01:08.545958 17707 ts_tablet_manager.cc:1996] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:08.547015 17707 tablet_replica.cc:333] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:08.547254 17707 raft_consensus.cc:2243] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:08.547468 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 5b2b8f6f05b74c849d19705a2e8ab0e7 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:08.547686 17707 raft_consensus.cc:2272] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.547735 16962 tablet_service.cc:1511] Processing CreateTablet for tablet 5e9b68ceb78f45edaaebaf7314364446 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.548229 16962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e9b68ceb78f45edaaebaf7314364446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.548197 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":84,"lbm_reads_lt_1ms":4}
I20260929 02:01:08.548789 17707 ts_tablet_manager.cc:1962] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.550041 17098 tablet_service.cc:1511] Processing CreateTablet for tablet 5e9b68ceb78f45edaaebaf7314364446 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.550513 17098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5e9b68ceb78f45edaaebaf7314364446. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.552966 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.553160 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.553666 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.356749
I20260929 02:01:08.554033 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.554747 17706 ts_tablet_manager.cc:1975] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.555023 17706 log.cc:1199] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/5b2b8f6f05b74c849d19705a2e8ab0e7
I20260929 02:01:08.555506 17706 ts_tablet_manager.cc:1996] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:08.556272 17707 ts_tablet_manager.cc:1975] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.556481 17707 log.cc:1199] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/925a0eec8efb496ba538b7f7c44d84f2
I20260929 02:01:08.556735 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 5b2b8f6f05b74c849d19705a2e8ab0e7 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:08.556939 17707 ts_tablet_manager.cc:1996] T 925a0eec8efb496ba538b7f7c44d84f2 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:08.557953 17707 tablet_replica.cc:333] T 5acedab1b27c414fa5181e35aa9ac24b P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:08.558097 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 925a0eec8efb496ba538b7f7c44d84f2 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:08.558246 17707 raft_consensus.cc:2243] T 5acedab1b27c414fa5181e35aa9ac24b P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:08.558584 17707 raft_consensus.cc:2272] T 5acedab1b27c414fa5181e35aa9ac24b P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.558694 17706 tablet_replica.cc:333] T 167d937681dc4e509f1d5d4ce2358d27 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:08.558988 17706 raft_consensus.cc:2243] T 167d937681dc4e509f1d5d4ce2358d27 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:08.559422 17707 ts_tablet_manager.cc:1962] T 5acedab1b27c414fa5181e35aa9ac24b P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.560204 17708 ts_tablet_manager.cc:1975] T a21bdeb06bb24a45945e00b68b17ca5b P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.560240 17706 raft_consensus.cc:2272] T 167d937681dc4e509f1d5d4ce2358d27 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.560420 17708 log.cc:1199] T a21bdeb06bb24a45945e00b68b17ca5b P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/a21bdeb06bb24a45945e00b68b17ca5b
I20260929 02:01:08.560902 17711 tablet_bootstrap.cc:492] T 5e9b68ceb78f45edaaebaf7314364446 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:08.560803 16953 tablet_service.cc:1511] Processing CreateTablet for tablet d57762935f614c4391a5236fd2cb72c4 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.561312 16953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d57762935f614c4391a5236fd2cb72c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.566359 17707 ts_tablet_manager.cc:1975] T 5acedab1b27c414fa5181e35aa9ac24b P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.566767 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.021s user 0.006s sys 0.005s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":1691,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":6844,"dirs.run_cpu_time_us":206,"dirs.run_wall_time_us":665,"lbm_read_time_us":531,"lbm_reads_lt_1ms":21,"lbm_write_time_us":324,"lbm_writes_lt_1ms":9,"mutex_wait_us":6746,"peak_mem_usage":32768,"reinsert_count":0,"update_count":16}
I20260929 02:01:08.567466 17706 ts_tablet_manager.cc:1962] T 167d937681dc4e509f1d5d4ce2358d27 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.567502 17707 log.cc:1199] T 5acedab1b27c414fa5181e35aa9ac24b P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/5acedab1b27c414fa5181e35aa9ac24b
I20260929 02:01:08.567956 17707 ts_tablet_manager.cc:1996] T 5acedab1b27c414fa5181e35aa9ac24b P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:08.569015 17707 tablet_replica.cc:333] T 167d937681dc4e509f1d5d4ce2358d27 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:08.569274 17707 raft_consensus.cc:2243] T 167d937681dc4e509f1d5d4ce2358d27 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:08.569504 17707 raft_consensus.cc:2272] T 167d937681dc4e509f1d5d4ce2358d27 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.569805 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 5acedab1b27c414fa5181e35aa9ac24b (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:08.570590 17707 ts_tablet_manager.cc:1962] T 167d937681dc4e509f1d5d4ce2358d27 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 02:01:08.570904 17186 consensus_peers.cc:597] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88 -> Peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Couldn't send request to peer 8668afa5b5ca4ebdabc812d0789de7bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5acedab1b27c414fa5181e35aa9ac24b. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:01:08.573660 17092 tablet_service.cc:1511] Processing CreateTablet for tablet d57762935f614c4391a5236fd2cb72c4 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.574081 17092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d57762935f614c4391a5236fd2cb72c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.575282 17218 tablet_service.cc:1511] Processing CreateTablet for tablet d57762935f614c4391a5236fd2cb72c4 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.575667 17218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet d57762935f614c4391a5236fd2cb72c4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.577903 16953 tablet_service.cc:1511] Processing CreateTablet for tablet c029d8ad8e534719be68303011ec7d8b (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.578328 17708 ts_tablet_manager.cc:1996] T a21bdeb06bb24a45945e00b68b17ca5b P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:08.578459 17707 ts_tablet_manager.cc:1975] T 167d937681dc4e509f1d5d4ce2358d27 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.578547 16953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c029d8ad8e534719be68303011ec7d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.578694 17707 log.cc:1199] T 167d937681dc4e509f1d5d4ce2358d27 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/167d937681dc4e509f1d5d4ce2358d27
I20260929 02:01:08.579326 17707 ts_tablet_manager.cc:1996] T 167d937681dc4e509f1d5d4ce2358d27 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:08.574270 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.071s user 0.017s sys 0.001s Metrics: {"bytes_written":711,"cfile_init":1,"delete_count":0,"dirs.queue_time_us":47,"dirs.run_cpu_time_us":158,"dirs.run_wall_time_us":786,"drs_written":1,"lbm_read_time_us":83,"lbm_reads_lt_1ms":4,"lbm_write_time_us":5559,"lbm_writes_1-10_ms":2,"lbm_writes_lt_1ms":29,"mutex_wait_us":31223,"peak_mem_usage":0,"reinsert_count":0,"rows_written":17,"update_count":16}
I20260929 02:01:08.579996 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.356749
I20260929 02:01:08.580632 17707 tablet_replica.cc:333] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:08.580655 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 167d937681dc4e509f1d5d4ce2358d27 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:08.580710 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260929 02:01:08.580807 17707 raft_consensus.cc:2243] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:08.580998 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.356749
I20260929 02:01:08.581141 17707 raft_consensus.cc:2272] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.581635 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":104,"lbm_reads_lt_1ms":4}
I20260929 02:01:08.581974 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.583063 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet a21bdeb06bb24a45945e00b68b17ca5b (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:08.583971 17708 tablet_replica.cc:333] T 925a0eec8efb496ba538b7f7c44d84f2 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:08.584249 16961 tablet_service.cc:1511] Processing CreateTablet for tablet e5aa0d7aa540479c89ae9b8b61d89c4e (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.584549 17594 tablet_bootstrap.cc:492] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:08.585126 17707 ts_tablet_manager.cc:1962] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.585131 16962 tablet_service.cc:1511] Processing CreateTablet for tablet b8b094aa976542c2bf9646de8377fe18 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.585443 17708 raft_consensus.cc:2243] T 925a0eec8efb496ba538b7f7c44d84f2 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:08.585810 16961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5aa0d7aa540479c89ae9b8b61d89c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.586891 17594 tablet_bootstrap.cc:654] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:08.588032 17711 tablet_bootstrap.cc:654] T 5e9b68ceb78f45edaaebaf7314364446 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:08.589275 16962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8b094aa976542c2bf9646de8377fe18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.592466 17707 ts_tablet_manager.cc:1975] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.592620 17707 log.cc:1199] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/c7892f83a6e34ba0899bb3ac41a749a5
I20260929 02:01:08.593232 16955 tablet_service.cc:1511] Processing CreateTablet for tablet 5d8486054d464e4ab45c10da4de28090 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.593360 16956 tablet_service.cc:1511] Processing CreateTablet for tablet fed81261ad7d4afabab4ada3e33aa5e2 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.593433 17707 ts_tablet_manager.cc:1996] T c7892f83a6e34ba0899bb3ac41a749a5 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:08.593703 16955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d8486054d464e4ab45c10da4de28090. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.594014 16956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fed81261ad7d4afabab4ada3e33aa5e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.594353 17707 tablet_replica.cc:333] T 306f1984e9324a949db3435c56fcfb8f P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:08.594548 17707 raft_consensus.cc:2243] T 306f1984e9324a949db3435c56fcfb8f P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:08.594714 17707 raft_consensus.cc:2272] T 306f1984e9324a949db3435c56fcfb8f P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.595960 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.092s user 0.006s sys 0.005s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":6425,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":381,"lbm_read_time_us":503,"lbm_reads_lt_1ms":27,"lbm_write_time_us":299,"lbm_writes_lt_1ms":9,"peak_mem_usage":32768,"reinsert_count":0,"update_count":18}
I20260929 02:01:08.597527 17098 tablet_service.cc:1511] Processing CreateTablet for tablet c029d8ad8e534719be68303011ec7d8b (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.597636 17092 tablet_service.cc:1511] Processing CreateTablet for tablet b8b094aa976542c2bf9646de8377fe18 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.597959 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet c7892f83a6e34ba0899bb3ac41a749a5 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:08.598079 17085 tablet_service.cc:1511] Processing CreateTablet for tablet 5d8486054d464e4ab45c10da4de28090 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.598362 17087 tablet_service.cc:1511] Processing CreateTablet for tablet fed81261ad7d4afabab4ada3e33aa5e2 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.598080 17095 tablet_service.cc:1511] Processing CreateTablet for tablet e5aa0d7aa540479c89ae9b8b61d89c4e (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.598455 17085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d8486054d464e4ab45c10da4de28090. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.599342 17098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c029d8ad8e534719be68303011ec7d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.599534 17087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fed81261ad7d4afabab4ada3e33aa5e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.600646 17092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8b094aa976542c2bf9646de8377fe18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.604660 17095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5aa0d7aa540479c89ae9b8b61d89c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.607018 17088 tablet_service.cc:1511] Processing CreateTablet for tablet 16931d61d19f4b1eaed27200e43c5336 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.607863 17088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16931d61d19f4b1eaed27200e43c5336. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.616088 17707 ts_tablet_manager.cc:1962] T 306f1984e9324a949db3435c56fcfb8f P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.618845 16956 tablet_service.cc:1511] Processing CreateTablet for tablet 16931d61d19f4b1eaed27200e43c5336 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.619637 16956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16931d61d19f4b1eaed27200e43c5336. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.585789 17708 raft_consensus.cc:2272] T 925a0eec8efb496ba538b7f7c44d84f2 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.624526 17708 ts_tablet_manager.cc:1962] T 925a0eec8efb496ba538b7f7c44d84f2 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.628968 17594 tablet_bootstrap.cc:492] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:08.629209 17594 ts_tablet_manager.cc:1449] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.045s user 0.011s sys 0.002s
I20260929 02:01:08.630429 17594 raft_consensus.cc:359] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.630810 17594 raft_consensus.cc:385] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:08.630956 17594 raft_consensus.cc:740] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:08.631315 17594 consensus_queue.cc:260] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.633127 17223 tablet_service.cc:1511] Processing CreateTablet for tablet c029d8ad8e534719be68303011ec7d8b (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.633566 17223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet c029d8ad8e534719be68303011ec7d8b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.634035 17222 tablet_service.cc:1511] Processing CreateTablet for tablet b8b094aa976542c2bf9646de8377fe18 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.634361 17222 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b8b094aa976542c2bf9646de8377fe18. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.635239 17594 ts_tablet_manager.cc:1480] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.006s user 0.002s sys 0.000s
I20260929 02:01:08.635555 17594 tablet_bootstrap.cc:492] T 5e9b68ceb78f45edaaebaf7314364446 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:08.635931 17118 raft_consensus.cc:3060] T 8f328eef30d541c7a61e7d7403e597c7 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:08.637887 16962 tablet_service.cc:1511] Processing CreateTablet for tablet e050cc7cbfa445e3be6081ca25c9ed63 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.638161 16953 tablet_service.cc:1511] Processing CreateTablet for tablet 2e8c4b957e16467fa6bd6e9c317b1b1f (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.638366 16962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e050cc7cbfa445e3be6081ca25c9ed63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.638720 16953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e8c4b957e16467fa6bd6e9c317b1b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.638931 16955 tablet_service.cc:1511] Processing CreateTablet for tablet 26ea597431574ea9a145a78687442f50 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.639504 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.059s user 0.013s sys 0.000s Metrics: {"cfile_cache_miss":5,"cfile_cache_miss_bytes":1691,"cfile_init":4,"delete_count":0,"delta_blocks_compacted":1,"delta_iterators_relevant":1,"dirs.queue_time_us":50,"dirs.run_cpu_time_us":132,"dirs.run_wall_time_us":1312,"lbm_read_time_us":447,"lbm_reads_lt_1ms":21,"lbm_write_time_us":341,"lbm_writes_lt_1ms":9,"mutex_wait_us":7686,"peak_mem_usage":32768,"reinsert_count":0,"update_count":16}
I20260929 02:01:08.640143 17219 tablet_service.cc:1511] Processing CreateTablet for tablet fed81261ad7d4afabab4ada3e33aa5e2 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.640743 17224 tablet_service.cc:1511] Processing CreateTablet for tablet 16931d61d19f4b1eaed27200e43c5336 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.641286 17231 tablet_service.cc:1511] Processing CreateTablet for tablet e050cc7cbfa445e3be6081ca25c9ed63 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.641865 17220 tablet_service.cc:1511] Processing CreateTablet for tablet e865b8f545e54b7eb0015f6bc1374540 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.641986 16955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26ea597431574ea9a145a78687442f50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.642380 17225 tablet_service.cc:1511] Processing CreateTablet for tablet 26ea597431574ea9a145a78687442f50 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.642442 16952 tablet_service.cc:1511] Processing CreateTablet for tablet 273fd95488284ef88c2272052f0de976 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.642812 16952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 273fd95488284ef88c2272052f0de976. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.642947 17221 tablet_service.cc:1511] Processing CreateTablet for tablet 2e8c4b957e16467fa6bd6e9c317b1b1f (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.643666 17219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet fed81261ad7d4afabab4ada3e33aa5e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.643592 16954 tablet_service.cc:1511] Processing CreateTablet for tablet 800999ad9c144b02b58cf1bf02b6d9e2 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.643987 16954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 800999ad9c144b02b58cf1bf02b6d9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.640709 16961 tablet_service.cc:1511] Processing CreateTablet for tablet e865b8f545e54b7eb0015f6bc1374540 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.645090 16956 tablet_service.cc:1511] Processing CreateTablet for tablet f8dd03b8a32f4ec5917415ee176d272b (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.649807 17092 tablet_service.cc:1511] Processing CreateTablet for tablet e865b8f545e54b7eb0015f6bc1374540 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.649998 17098 tablet_service.cc:1511] Processing CreateTablet for tablet 26ea597431574ea9a145a78687442f50 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.650210 17092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e865b8f545e54b7eb0015f6bc1374540. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.650525 17095 tablet_service.cc:1511] Processing CreateTablet for tablet 2e8c4b957e16467fa6bd6e9c317b1b1f (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.651495 17232 tablet_service.cc:1511] Processing CreateTablet for tablet 5d8486054d464e4ab45c10da4de28090 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.653743 17095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e8c4b957e16467fa6bd6e9c317b1b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.654389 17096 tablet_service.cc:1511] Processing CreateTablet for tablet 273fd95488284ef88c2272052f0de976 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.656862 17229 tablet_service.cc:1511] Processing CreateTablet for tablet e5aa0d7aa540479c89ae9b8b61d89c4e (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.657469 17225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26ea597431574ea9a145a78687442f50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.658499 17228 tablet_service.cc:1511] Processing CreateTablet for tablet 800999ad9c144b02b58cf1bf02b6d9e2 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.659495 17097 tablet_service.cc:1511] Processing CreateTablet for tablet 964849b43d28439aa45e8a51084bcae9 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.661597 17233 tablet_service.cc:1511] Processing CreateTablet for tablet 273fd95488284ef88c2272052f0de976 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.663508 17221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 2e8c4b957e16467fa6bd6e9c317b1b1f. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.665006 17231 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e050cc7cbfa445e3be6081ca25c9ed63. 1 dirs total, 0 dirs full, 0 dirs failed
W20260929 02:01:08.666714 16919 consensus_peers.cc:597] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f -> Peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Couldn't send request to peer 8668afa5b5ca4ebdabc812d0789de7bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b2b8f6f05b74c849d19705a2e8ab0e7. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:01:08.666872 17220 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e865b8f545e54b7eb0015f6bc1374540. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.666626 16964 tablet_service.cc:1511] Processing CreateTablet for tablet 964849b43d28439aa45e8a51084bcae9 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.666880 17226 tablet_service.cc:1511] Processing CreateTablet for tablet 964849b43d28439aa45e8a51084bcae9 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.666628 17623 consensus_queue.cc:1048] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.001s
I20260929 02:01:08.667332 17226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 964849b43d28439aa45e8a51084bcae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.669438 17594 tablet_bootstrap.cc:654] T 5e9b68ceb78f45edaaebaf7314364446 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:08.670495 17098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 26ea597431574ea9a145a78687442f50. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.671090 17097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 964849b43d28439aa45e8a51084bcae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.671448 17090 tablet_service.cc:1511] Processing CreateTablet for tablet f8dd03b8a32f4ec5917415ee176d272b (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.672133 17096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 273fd95488284ef88c2272052f0de976. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.673053 17233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 273fd95488284ef88c2272052f0de976. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.674033 17087 tablet_service.cc:1511] Processing CreateTablet for tablet e050cc7cbfa445e3be6081ca25c9ed63 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.677044 17090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8dd03b8a32f4ec5917415ee176d272b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.677145 17232 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5d8486054d464e4ab45c10da4de28090. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.677409 17229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e5aa0d7aa540479c89ae9b8b61d89c4e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.677415 17087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e050cc7cbfa445e3be6081ca25c9ed63. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.678570 17224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 16931d61d19f4b1eaed27200e43c5336. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.679584 17706 ts_tablet_manager.cc:1975] T 167d937681dc4e509f1d5d4ce2358d27 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.679719 17706 log.cc:1199] T 167d937681dc4e509f1d5d4ce2358d27 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/167d937681dc4e509f1d5d4ce2358d27
I20260929 02:01:08.680200 17706 ts_tablet_manager.cc:1996] T 167d937681dc4e509f1d5d4ce2358d27 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:08.684988 17218 tablet_service.cc:1511] Processing CreateTablet for tablet f8dd03b8a32f4ec5917415ee176d272b (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.687546 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 167d937681dc4e509f1d5d4ce2358d27 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:08.670228 17085 tablet_service.cc:1511] Processing CreateTablet for tablet 800999ad9c144b02b58cf1bf02b6d9e2 (DEFAULT_TABLE table=table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:08.691804 17228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 800999ad9c144b02b58cf1bf02b6d9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.705381 17706 tablet_replica.cc:333] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:08.706950 16961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e865b8f545e54b7eb0015f6bc1374540. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.710661 17218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8dd03b8a32f4ec5917415ee176d272b. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.710698 17711 tablet_bootstrap.cc:492] T 5e9b68ceb78f45edaaebaf7314364446 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:08.710867 17711 ts_tablet_manager.cc:1449] T 5e9b68ceb78f45edaaebaf7314364446 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.150s user 0.012s sys 0.007s
I20260929 02:01:08.710983 17706 raft_consensus.cc:2243] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:08.711346 17706 raft_consensus.cc:2272] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.711740 17711 raft_consensus.cc:359] T 5e9b68ceb78f45edaaebaf7314364446 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } } peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } }
I20260929 02:01:08.711915 17711 raft_consensus.cc:385] T 5e9b68ceb78f45edaaebaf7314364446 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:08.711988 17711 raft_consensus.cc:740] T 5e9b68ceb78f45edaaebaf7314364446 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:08.712213 17711 consensus_queue.cc:260] T 5e9b68ceb78f45edaaebaf7314364446 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } } peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } }
I20260929 02:01:08.712747 17711 ts_tablet_manager.cc:1480] T 5e9b68ceb78f45edaaebaf7314364446 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260929 02:01:08.713689 16964 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 964849b43d28439aa45e8a51084bcae9. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.718073 17595 tablet_bootstrap.cc:492] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:08.718770 17085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 800999ad9c144b02b58cf1bf02b6d9e2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:08.719893 17337 consensus_queue.cc:1048] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:08.727427 17706 ts_tablet_manager.cc:1962] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.733309 17595 ts_tablet_manager.cc:1449] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.180s user 0.014s sys 0.030s
I20260929 02:01:08.734436 17595 raft_consensus.cc:359] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } } peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } }
I20260929 02:01:08.734748 17595 raft_consensus.cc:385] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:08.734858 17595 raft_consensus.cc:740] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
W20260929 02:01:08.735522 16919 consensus_peers.cc:597] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f -> Peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Couldn't send request to peer 8668afa5b5ca4ebdabc812d0789de7bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 167d937681dc4e509f1d5d4ce2358d27. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:01:08.735908 17595 consensus_queue.cc:260] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } } peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } }
I20260929 02:01:08.737244 17595 ts_tablet_manager.cc:1480] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260929 02:01:08.737532 17595 tablet_bootstrap.cc:492] T d57762935f614c4391a5236fd2cb72c4 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:08.747244 17711 tablet_bootstrap.cc:492] T d57762935f614c4391a5236fd2cb72c4 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:08.748805 16956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet f8dd03b8a32f4ec5917415ee176d272b. 1 dirs total, 0 dirs full, 0 dirs failed
W20260929 02:01:08.755223 16921 consensus_peers.cc:597] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f -> Peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Couldn't send request to peer d103dd4823044faebe46100a0dfbee88. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 5b2b8f6f05b74c849d19705a2e8ab0e7. This is attempt 1: this message will repeat every 5th retry.
W20260929 02:01:08.758487 16921 consensus_peers.cc:597] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f -> Peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Couldn't send request to peer d103dd4823044faebe46100a0dfbee88. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 167d937681dc4e509f1d5d4ce2358d27. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:01:08.770898 17711 tablet_bootstrap.cc:654] T d57762935f614c4391a5236fd2cb72c4 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:08.774523 17674 raft_consensus.cc:493] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:08.774701 17674 raft_consensus.cc:515] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } } peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } }
I20260929 02:01:08.775398 17674 leader_election.cc:290] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:08.776226 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e9b68ceb78f45edaaebaf7314364446" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:08.776257 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e9b68ceb78f45edaaebaf7314364446" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:08.776472 16984 raft_consensus.cc:2468] T 5e9b68ceb78f45edaaebaf7314364446 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 0.
W20260929 02:01:08.776808 17186 leader_election.cc:343] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:08.777233 17189 leader_election.cc:304] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc
I20260929 02:01:08.777725 17674 raft_consensus.cc:2804] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:08.777835 17674 raft_consensus.cc:493] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:08.777989 17674 raft_consensus.cc:3060] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:08.779126 17707 ts_tablet_manager.cc:1975] T 306f1984e9324a949db3435c56fcfb8f P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.779276 17707 log.cc:1199] T 306f1984e9324a949db3435c56fcfb8f P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/306f1984e9324a949db3435c56fcfb8f
I20260929 02:01:08.779891 17707 ts_tablet_manager.cc:1996] T 306f1984e9324a949db3435c56fcfb8f P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:08.780347 17595 tablet_bootstrap.cc:654] T d57762935f614c4391a5236fd2cb72c4 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:08.781153 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 306f1984e9324a949db3435c56fcfb8f (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:08.781464 17674 raft_consensus.cc:515] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } } peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } }
I20260929 02:01:08.783051 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e9b68ceb78f45edaaebaf7314364446" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:08.783262 16984 raft_consensus.cc:3060] T 5e9b68ceb78f45edaaebaf7314364446 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:08.783622 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e9b68ceb78f45edaaebaf7314364446" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
W20260929 02:01:08.784225 17186 leader_election.cc:343] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:08.784410 17674 leader_election.cc:290] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:08.786381 16984 raft_consensus.cc:2468] T 5e9b68ceb78f45edaaebaf7314364446 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 1.
I20260929 02:01:08.786860 17189 leader_election.cc:304] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc
I20260929 02:01:08.787082 17708 ts_tablet_manager.cc:1975] T 925a0eec8efb496ba538b7f7c44d84f2 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.787209 17707 tablet_replica.cc:333] T bbdc98a938b247bdaf113dc94c3b4d59 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:08.787262 17708 log.cc:1199] T 925a0eec8efb496ba538b7f7c44d84f2 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/925a0eec8efb496ba538b7f7c44d84f2
I20260929 02:01:08.787452 17707 raft_consensus.cc:2243] T bbdc98a938b247bdaf113dc94c3b4d59 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:08.787452 17674 raft_consensus.cc:2804] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:08.787642 17674 raft_consensus.cc:697] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Becoming Leader. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Running, Role: LEADER
I20260929 02:01:08.787873 17708 ts_tablet_manager.cc:1996] T 925a0eec8efb496ba538b7f7c44d84f2 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:08.788017 17674 consensus_queue.cc:237] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } } peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } }
I20260929 02:01:08.788913 17594 tablet_bootstrap.cc:492] T 5e9b68ceb78f45edaaebaf7314364446 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:08.789143 17707 raft_consensus.cc:2272] T bbdc98a938b247bdaf113dc94c3b4d59 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.789372 17594 ts_tablet_manager.cc:1449] T 5e9b68ceb78f45edaaebaf7314364446 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.154s user 0.001s sys 0.004s
I20260929 02:01:08.790392 17594 raft_consensus.cc:359] T 5e9b68ceb78f45edaaebaf7314364446 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } } peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } }
I20260929 02:01:08.790670 17594 raft_consensus.cc:385] T 5e9b68ceb78f45edaaebaf7314364446 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:08.790794 17594 raft_consensus.cc:740] T 5e9b68ceb78f45edaaebaf7314364446 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:08.791116 17594 consensus_queue.cc:260] T 5e9b68ceb78f45edaaebaf7314364446 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } } peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } }
I20260929 02:01:08.791821 17706 ts_tablet_manager.cc:1975] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.791898 17594 ts_tablet_manager.cc:1480] T 5e9b68ceb78f45edaaebaf7314364446 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 02:01:08.791985 17706 log.cc:1199] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/5acedab1b27c414fa5181e35aa9ac24b
I20260929 02:01:08.792428 17706 ts_tablet_manager.cc:1996] T 5acedab1b27c414fa5181e35aa9ac24b P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:08.792326 17594 tablet_bootstrap.cc:492] T e5aa0d7aa540479c89ae9b8b61d89c4e P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:08.793416 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 5acedab1b27c414fa5181e35aa9ac24b (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:08.795150 17594 tablet_bootstrap.cc:654] T e5aa0d7aa540479c89ae9b8b61d89c4e P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:08.800650 17706 tablet_replica.cc:333] T c6cac2e61f6f4795869af850549096a9 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:08.800913 17706 raft_consensus.cc:2243] T c6cac2e61f6f4795869af850549096a9 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:08.800921 17707 ts_tablet_manager.cc:1962] T bbdc98a938b247bdaf113dc94c3b4d59 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.801141 17706 raft_consensus.cc:2272] T c6cac2e61f6f4795869af850549096a9 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.803139 17706 ts_tablet_manager.cc:1962] T c6cac2e61f6f4795869af850549096a9 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 02:01:08.803341 17186 consensus_peers.cc:597] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88 -> Peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Couldn't send request to peer 8668afa5b5ca4ebdabc812d0789de7bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 306f1984e9324a949db3435c56fcfb8f. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:01:08.806749 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 925a0eec8efb496ba538b7f7c44d84f2 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:08.807891 17711 tablet_bootstrap.cc:492] T d57762935f614c4391a5236fd2cb72c4 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:08.808166 17711 ts_tablet_manager.cc:1449] T d57762935f614c4391a5236fd2cb72c4 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.070s user 0.003s sys 0.001s
I20260929 02:01:08.809118 17708 tablet_replica.cc:333] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:08.810810 17711 raft_consensus.cc:359] T d57762935f614c4391a5236fd2cb72c4 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.811427 17711 raft_consensus.cc:385] T d57762935f614c4391a5236fd2cb72c4 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:08.813410 17711 raft_consensus.cc:740] T d57762935f614c4391a5236fd2cb72c4 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:08.813805 17711 consensus_queue.cc:260] T d57762935f614c4391a5236fd2cb72c4 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.811565 17708 raft_consensus.cc:2243] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:08.814633 17595 tablet_bootstrap.cc:492] T d57762935f614c4391a5236fd2cb72c4 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:08.814791 17595 ts_tablet_manager.cc:1449] T d57762935f614c4391a5236fd2cb72c4 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.077s user 0.006s sys 0.002s
I20260929 02:01:08.815332 17711 ts_tablet_manager.cc:1480] T d57762935f614c4391a5236fd2cb72c4 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260929 02:01:08.816426 17708 raft_consensus.cc:2272] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.816399 17595 raft_consensus.cc:359] T d57762935f614c4391a5236fd2cb72c4 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.816690 17595 raft_consensus.cc:385] T d57762935f614c4391a5236fd2cb72c4 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:08.816790 17595 raft_consensus.cc:740] T d57762935f614c4391a5236fd2cb72c4 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:08.817057 17595 consensus_queue.cc:260] T d57762935f614c4391a5236fd2cb72c4 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.817421 17708 ts_tablet_manager.cc:1962] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.817540 17595 ts_tablet_manager.cc:1480] T d57762935f614c4391a5236fd2cb72c4 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:01:08.817390 16818 catalog_manager.cc:5838] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88 reported cstate change: term changed from 0 to 1, leader changed from <none> to d103dd4823044faebe46100a0dfbee88 (127.0.71.67). New cstate: current_term: 1 leader_uuid: "d103dd4823044faebe46100a0dfbee88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:08.817799 17595 tablet_bootstrap.cc:492] T b8b094aa976542c2bf9646de8377fe18 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:08.819566 17595 tablet_bootstrap.cc:654] T b8b094aa976542c2bf9646de8377fe18 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:08.824409 17711 tablet_bootstrap.cc:492] T c029d8ad8e534719be68303011ec7d8b P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:08.826076 17711 tablet_bootstrap.cc:654] T c029d8ad8e534719be68303011ec7d8b P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:08.828805 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.829013 17594 tablet_bootstrap.cc:492] T e5aa0d7aa540479c89ae9b8b61d89c4e P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:08.829195 17594 ts_tablet_manager.cc:1449] T e5aa0d7aa540479c89ae9b8b61d89c4e P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.037s user 0.001s sys 0.002s
I20260929 02:01:08.829169 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.829572 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.829630 17706 ts_tablet_manager.cc:1975] T c6cac2e61f6f4795869af850549096a9 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.829770 17706 log.cc:1199] T c6cac2e61f6f4795869af850549096a9 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/c6cac2e61f6f4795869af850549096a9
I20260929 02:01:08.829897 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.830215 17706 ts_tablet_manager.cc:1996] T c6cac2e61f6f4795869af850549096a9 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:08.830224 17594 raft_consensus.cc:359] T e5aa0d7aa540479c89ae9b8b61d89c4e P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.830442 17594 raft_consensus.cc:385] T e5aa0d7aa540479c89ae9b8b61d89c4e P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:08.830551 17594 raft_consensus.cc:740] T e5aa0d7aa540479c89ae9b8b61d89c4e P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:08.830847 17594 consensus_queue.cc:260] T e5aa0d7aa540479c89ae9b8b61d89c4e P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.831321 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet c6cac2e61f6f4795869af850549096a9 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:08.831656 17706 tablet_replica.cc:333] T ae7f6651c994403697a0f04508b70dae P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:08.831784 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.831871 17706 raft_consensus.cc:2243] T ae7f6651c994403697a0f04508b70dae P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:08.832041 17706 raft_consensus.cc:2272] T ae7f6651c994403697a0f04508b70dae P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.832085 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.832322 17708 ts_tablet_manager.cc:1975] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.832331 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.832475 17708 log.cc:1199] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/5b2b8f6f05b74c849d19705a2e8ab0e7
I20260929 02:01:08.832870 17708 ts_tablet_manager.cc:1996] T 5b2b8f6f05b74c849d19705a2e8ab0e7 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:08.833355 17595 tablet_bootstrap.cc:492] T b8b094aa976542c2bf9646de8377fe18 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:08.833523 17595 ts_tablet_manager.cc:1449] T b8b094aa976542c2bf9646de8377fe18 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.001s
I20260929 02:01:08.833932 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 5b2b8f6f05b74c849d19705a2e8ab0e7 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:08.834980 17595 raft_consensus.cc:359] T b8b094aa976542c2bf9646de8377fe18 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.835134 17594 ts_tablet_manager.cc:1480] T e5aa0d7aa540479c89ae9b8b61d89c4e P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.006s user 0.003s sys 0.000s
I20260929 02:01:08.835165 17708 tablet_replica.cc:333] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:08.835177 17595 raft_consensus.cc:385] T b8b094aa976542c2bf9646de8377fe18 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:08.835244 17595 raft_consensus.cc:740] T b8b094aa976542c2bf9646de8377fe18 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:08.835372 17708 raft_consensus.cc:2243] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:08.835485 17595 consensus_queue.cc:260] T b8b094aa976542c2bf9646de8377fe18 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.835711 17708 raft_consensus.cc:2272] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.837191 17595 ts_tablet_manager.cc:1480] T b8b094aa976542c2bf9646de8377fe18 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260929 02:01:08.837595 17595 tablet_bootstrap.cc:492] T 5d8486054d464e4ab45c10da4de28090 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:08.838339 17594 tablet_bootstrap.cc:492] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:08.839986 17595 tablet_bootstrap.cc:654] T 5d8486054d464e4ab45c10da4de28090 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:08.840236 17594 tablet_bootstrap.cc:654] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:08.841145 17706 ts_tablet_manager.cc:1962] T ae7f6651c994403697a0f04508b70dae P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.841395 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.841734 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.842018 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.842278 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.844173 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.844528 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.844806 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.845323 17707 ts_tablet_manager.cc:1975] T bbdc98a938b247bdaf113dc94c3b4d59 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.845793 17707 log.cc:1199] T bbdc98a938b247bdaf113dc94c3b4d59 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/bbdc98a938b247bdaf113dc94c3b4d59
I20260929 02:01:08.846741 17707 ts_tablet_manager.cc:1996] T bbdc98a938b247bdaf113dc94c3b4d59 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:08.847618 17706 ts_tablet_manager.cc:1975] T ae7f6651c994403697a0f04508b70dae P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.847762 17706 log.cc:1199] T ae7f6651c994403697a0f04508b70dae P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/ae7f6651c994403697a0f04508b70dae
I20260929 02:01:08.848157 17706 ts_tablet_manager.cc:1996] T ae7f6651c994403697a0f04508b70dae P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:08.849223 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet bbdc98a938b247bdaf113dc94c3b4d59 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:08.846179 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.016s user 0.004s sys 0.001s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":170,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:08.853386 17707 tablet_replica.cc:333] T 7239b23c6e30483d9bd0738e07c63610 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:08.853621 17707 raft_consensus.cc:2243] T 7239b23c6e30483d9bd0738e07c63610 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:08.853830 17707 raft_consensus.cc:2272] T 7239b23c6e30483d9bd0738e07c63610 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.854534 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.012s user 0.004s sys 0.001s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":177,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:08.854741 17707 ts_tablet_manager.cc:1962] T 7239b23c6e30483d9bd0738e07c63610 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.856842 17711 tablet_bootstrap.cc:492] T c029d8ad8e534719be68303011ec7d8b P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:08.857044 17711 ts_tablet_manager.cc:1449] T c029d8ad8e534719be68303011ec7d8b P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.033s user 0.004s sys 0.006s
I20260929 02:01:08.857813 17711 raft_consensus.cc:359] T c029d8ad8e534719be68303011ec7d8b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.858059 17711 raft_consensus.cc:385] T c029d8ad8e534719be68303011ec7d8b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:08.858186 17711 raft_consensus.cc:740] T c029d8ad8e534719be68303011ec7d8b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:08.858510 17711 consensus_queue.cc:260] T c029d8ad8e534719be68303011ec7d8b P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.861596 17711 ts_tablet_manager.cc:1480] T c029d8ad8e534719be68303011ec7d8b P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.004s user 0.001s sys 0.000s
I20260929 02:01:08.861764 17708 ts_tablet_manager.cc:1962] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.863868 17707 ts_tablet_manager.cc:1975] T 7239b23c6e30483d9bd0738e07c63610 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.864007 17594 tablet_bootstrap.cc:492] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:08.864032 17707 log.cc:1199] T 7239b23c6e30483d9bd0738e07c63610 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/7239b23c6e30483d9bd0738e07c63610
I20260929 02:01:08.864197 17594 ts_tablet_manager.cc:1449] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.000s
I20260929 02:01:08.864589 17707 ts_tablet_manager.cc:1996] T 7239b23c6e30483d9bd0738e07c63610 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:08.865661 17711 tablet_bootstrap.cc:492] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:08.865744 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 7239b23c6e30483d9bd0738e07c63610 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:08.866169 17707 tablet_replica.cc:333] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:08.866214 17594 raft_consensus.cc:359] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.866426 17707 raft_consensus.cc:2243] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:08.866837 17707 raft_consensus.cc:2272] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.866456 17594 raft_consensus.cc:385] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:08.867074 17594 raft_consensus.cc:740] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:08.867350 17594 consensus_queue.cc:260] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.868507 17711 tablet_bootstrap.cc:654] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:08.871141 17594 ts_tablet_manager.cc:1480] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20260929 02:01:08.873677 17595 tablet_bootstrap.cc:492] T 5d8486054d464e4ab45c10da4de28090 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:08.873878 17595 ts_tablet_manager.cc:1449] T 5d8486054d464e4ab45c10da4de28090 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.036s user 0.004s sys 0.000s
I20260929 02:01:08.874342 17594 tablet_bootstrap.cc:492] T b8b094aa976542c2bf9646de8377fe18 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:08.874902 17595 raft_consensus.cc:359] T 5d8486054d464e4ab45c10da4de28090 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.875154 17595 raft_consensus.cc:385] T 5d8486054d464e4ab45c10da4de28090 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:08.875244 17595 raft_consensus.cc:740] T 5d8486054d464e4ab45c10da4de28090 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:08.875552 17595 consensus_queue.cc:260] T 5d8486054d464e4ab45c10da4de28090 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.876472 17594 tablet_bootstrap.cc:654] T b8b094aa976542c2bf9646de8377fe18 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:08.876663 17595 ts_tablet_manager.cc:1480] T 5d8486054d464e4ab45c10da4de28090 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 02:01:08.876995 17595 tablet_bootstrap.cc:492] T e865b8f545e54b7eb0015f6bc1374540 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:08.877755 17711 tablet_bootstrap.cc:492] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:08.878000 17711 ts_tablet_manager.cc:1449] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.012s user 0.002s sys 0.008s
I20260929 02:01:08.878827 17595 tablet_bootstrap.cc:654] T e865b8f545e54b7eb0015f6bc1374540 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:08.878865 17711 raft_consensus.cc:359] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.879186 17711 raft_consensus.cc:385] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:08.879307 17711 raft_consensus.cc:740] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:08.879662 17711 consensus_queue.cc:260] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.880394 17706 tablet_replica.cc:333] T c7892f83a6e34ba0899bb3ac41a749a5 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:08.880596 17706 raft_consensus.cc:2243] T c7892f83a6e34ba0899bb3ac41a749a5 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:08.880784 17706 raft_consensus.cc:2272] T c7892f83a6e34ba0899bb3ac41a749a5 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.881095 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet ae7f6651c994403697a0f04508b70dae (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:08.881794 17707 ts_tablet_manager.cc:1962] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.882809 17708 ts_tablet_manager.cc:1975] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.883026 17708 log.cc:1199] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/167d937681dc4e509f1d5d4ce2358d27
I20260929 02:01:08.883513 17708 ts_tablet_manager.cc:1996] T 167d937681dc4e509f1d5d4ce2358d27 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:08.883646 17706 ts_tablet_manager.cc:1962] T c7892f83a6e34ba0899bb3ac41a749a5 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.891192 17706 ts_tablet_manager.cc:1975] T c7892f83a6e34ba0899bb3ac41a749a5 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.891330 17706 log.cc:1199] T c7892f83a6e34ba0899bb3ac41a749a5 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/c7892f83a6e34ba0899bb3ac41a749a5
I20260929 02:01:08.891763 17706 ts_tablet_manager.cc:1996] T c7892f83a6e34ba0899bb3ac41a749a5 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:08.893007 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet c7892f83a6e34ba0899bb3ac41a749a5 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:08.893059 17707 ts_tablet_manager.cc:1975] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.893181 17707 log.cc:1199] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/c6cac2e61f6f4795869af850549096a9
I20260929 02:01:08.893479 17708 tablet_replica.cc:333] T d9511e2734f1414ba6df2e7c9459ce34 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:08.893553 17706 tablet_replica.cc:333] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:08.893519 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 167d937681dc4e509f1d5d4ce2358d27 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:08.893695 17708 raft_consensus.cc:2243] T d9511e2734f1414ba6df2e7c9459ce34 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:08.893774 17706 raft_consensus.cc:2243] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:08.893911 17708 raft_consensus.cc:2272] T d9511e2734f1414ba6df2e7c9459ce34 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.894109 17706 raft_consensus.cc:2272] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.894572 17708 ts_tablet_manager.cc:1962] T d9511e2734f1414ba6df2e7c9459ce34 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.894672 17594 tablet_bootstrap.cc:492] T b8b094aa976542c2bf9646de8377fe18 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:08.894861 17594 ts_tablet_manager.cc:1449] T b8b094aa976542c2bf9646de8377fe18 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.021s user 0.003s sys 0.001s
I20260929 02:01:08.894976 17706 ts_tablet_manager.cc:1962] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.895676 17594 raft_consensus.cc:359] T b8b094aa976542c2bf9646de8377fe18 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.895900 17594 raft_consensus.cc:385] T b8b094aa976542c2bf9646de8377fe18 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:08.896004 17594 raft_consensus.cc:740] T b8b094aa976542c2bf9646de8377fe18 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:08.896271 17594 consensus_queue.cc:260] T b8b094aa976542c2bf9646de8377fe18 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.883052 17711 ts_tablet_manager.cc:1480] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260929 02:01:08.896835 17711 tablet_bootstrap.cc:492] T fed81261ad7d4afabab4ada3e33aa5e2 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:08.899067 17711 tablet_bootstrap.cc:654] T fed81261ad7d4afabab4ada3e33aa5e2 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:08.900427 17594 ts_tablet_manager.cc:1480] T b8b094aa976542c2bf9646de8377fe18 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260929 02:01:08.901326 17594 tablet_bootstrap.cc:492] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:08.903811 17594 tablet_bootstrap.cc:654] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:08.904309 17595 tablet_bootstrap.cc:492] T e865b8f545e54b7eb0015f6bc1374540 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:08.904517 17595 ts_tablet_manager.cc:1449] T e865b8f545e54b7eb0015f6bc1374540 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.028s user 0.001s sys 0.003s
I20260929 02:01:08.905084 17707 ts_tablet_manager.cc:1996] T c6cac2e61f6f4795869af850549096a9 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:08.905296 17595 raft_consensus.cc:359] T e865b8f545e54b7eb0015f6bc1374540 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.905539 17595 raft_consensus.cc:385] T e865b8f545e54b7eb0015f6bc1374540 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:08.905632 17595 raft_consensus.cc:740] T e865b8f545e54b7eb0015f6bc1374540 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:08.906059 17595 consensus_queue.cc:260] T e865b8f545e54b7eb0015f6bc1374540 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.906596 17595 ts_tablet_manager.cc:1480] T e865b8f545e54b7eb0015f6bc1374540 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.002s user 0.001s sys 0.000s
I20260929 02:01:08.906870 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet c6cac2e61f6f4795869af850549096a9 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:08.906915 17595 tablet_bootstrap.cc:492] T e050cc7cbfa445e3be6081ca25c9ed63 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:08.907415 17707 tablet_replica.cc:333] T ae7f6651c994403697a0f04508b70dae P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:08.907876 17707 raft_consensus.cc:2243] T ae7f6651c994403697a0f04508b70dae P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:08.908085 17707 raft_consensus.cc:2272] T ae7f6651c994403697a0f04508b70dae P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.908850 17595 tablet_bootstrap.cc:654] T e050cc7cbfa445e3be6081ca25c9ed63 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:08.909159 17543 raft_consensus.cc:493] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:08.909339 17543 raft_consensus.cc:515] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.910048 17543 leader_election.cc:290] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:08.911007 17250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d57762935f614c4391a5236fd2cb72c4" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:08.911098 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d57762935f614c4391a5236fd2cb72c4" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:08.911231 17250 raft_consensus.cc:2468] T d57762935f614c4391a5236fd2cb72c4 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:08.911387 16984 raft_consensus.cc:2468] T d57762935f614c4391a5236fd2cb72c4 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:08.911972 17056 leader_election.cc:304] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:08.912343 17595 tablet_bootstrap.cc:492] T e050cc7cbfa445e3be6081ca25c9ed63 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:08.912395 17543 raft_consensus.cc:2804] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:08.912508 17543 raft_consensus.cc:493] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:08.912531 17595 ts_tablet_manager.cc:1449] T e050cc7cbfa445e3be6081ca25c9ed63 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.006s user 0.004s sys 0.000s
I20260929 02:01:08.912616 17543 raft_consensus.cc:3060] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:08.912540 17706 ts_tablet_manager.cc:1975] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.913303 17711 tablet_bootstrap.cc:492] T fed81261ad7d4afabab4ada3e33aa5e2 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:08.913403 17711 ts_tablet_manager.cc:1449] T fed81261ad7d4afabab4ada3e33aa5e2 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.017s user 0.002s sys 0.002s
I20260929 02:01:08.914101 17711 raft_consensus.cc:359] T fed81261ad7d4afabab4ada3e33aa5e2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.914264 17711 raft_consensus.cc:385] T fed81261ad7d4afabab4ada3e33aa5e2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:08.914336 17711 raft_consensus.cc:740] T fed81261ad7d4afabab4ada3e33aa5e2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:08.914593 17711 consensus_queue.cc:260] T fed81261ad7d4afabab4ada3e33aa5e2 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.915127 17711 ts_tablet_manager.cc:1480] T fed81261ad7d4afabab4ada3e33aa5e2 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 02:01:08.915617 17706 log.cc:1199] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/306f1984e9324a949db3435c56fcfb8f
I20260929 02:01:08.916375 17708 ts_tablet_manager.cc:1975] T d9511e2734f1414ba6df2e7c9459ce34 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.916507 17708 log.cc:1199] T d9511e2734f1414ba6df2e7c9459ce34 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/d9511e2734f1414ba6df2e7c9459ce34
I20260929 02:01:08.917035 17708 ts_tablet_manager.cc:1996] T d9511e2734f1414ba6df2e7c9459ce34 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:08.917775 17711 tablet_bootstrap.cc:492] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:08.918326 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet d9511e2734f1414ba6df2e7c9459ce34 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:08.919126 17595 raft_consensus.cc:359] T e050cc7cbfa445e3be6081ca25c9ed63 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.919909 17708 tablet_replica.cc:333] T 5acedab1b27c414fa5181e35aa9ac24b P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:08.920051 17595 raft_consensus.cc:385] T e050cc7cbfa445e3be6081ca25c9ed63 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:08.920367 17708 raft_consensus.cc:2243] T 5acedab1b27c414fa5181e35aa9ac24b P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:08.920562 17595 raft_consensus.cc:740] T e050cc7cbfa445e3be6081ca25c9ed63 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:08.920692 17706 ts_tablet_manager.cc:1996] T 306f1984e9324a949db3435c56fcfb8f P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:08.921087 17707 ts_tablet_manager.cc:1962] T ae7f6651c994403697a0f04508b70dae P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.921015 17711 tablet_bootstrap.cc:654] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:08.921365 17595 consensus_queue.cc:260] T e050cc7cbfa445e3be6081ca25c9ed63 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.922235 17708 raft_consensus.cc:2272] T 5acedab1b27c414fa5181e35aa9ac24b P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.922552 17543 raft_consensus.cc:515] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.923667 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 306f1984e9324a949db3435c56fcfb8f (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:08.924515 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d57762935f614c4391a5236fd2cb72c4" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:08.924731 16984 raft_consensus.cc:3060] T d57762935f614c4391a5236fd2cb72c4 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:08.925530 17708 ts_tablet_manager.cc:1962] T 5acedab1b27c414fa5181e35aa9ac24b P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.927532 17595 ts_tablet_manager.cc:1480] T e050cc7cbfa445e3be6081ca25c9ed63 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.015s user 0.002s sys 0.000s
I20260929 02:01:08.927860 17706 tablet_replica.cc:333] T bbdc98a938b247bdaf113dc94c3b4d59 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:08.928110 17706 raft_consensus.cc:2243] T bbdc98a938b247bdaf113dc94c3b4d59 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:08.928342 17706 raft_consensus.cc:2272] T bbdc98a938b247bdaf113dc94c3b4d59 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.929664 16984 raft_consensus.cc:2468] T d57762935f614c4391a5236fd2cb72c4 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:08.930147 17056 leader_election.cc:304] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:08.930552 17398 raft_consensus.cc:2804] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:08.930716 17398 raft_consensus.cc:697] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Becoming Leader. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Running, Role: LEADER
I20260929 02:01:08.930905 17595 tablet_bootstrap.cc:492] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:08.931053 17398 consensus_queue.cc:237] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.933391 17595 tablet_bootstrap.cc:654] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:08.934069 17543 leader_election.cc:290] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:08.934752 16819 catalog_manager.cc:5838] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66). New cstate: current_term: 1 leader_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:08.935305 17250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "d57762935f614c4391a5236fd2cb72c4" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:08.935556 17250 raft_consensus.cc:3060] T d57762935f614c4391a5236fd2cb72c4 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:08.939864 17706 ts_tablet_manager.cc:1962] T bbdc98a938b247bdaf113dc94c3b4d59 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.939991 17250 raft_consensus.cc:2468] T d57762935f614c4391a5236fd2cb72c4 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:08.948314 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.948693 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.948997 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.949268 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.950989 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.951272 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.951511 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.951905 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.212476
I20260929 02:01:08.952169 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.212476
I20260929 02:01:08.952409 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.212476
I20260929 02:01:08.952731 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.952811 17706 ts_tablet_manager.cc:1975] T bbdc98a938b247bdaf113dc94c3b4d59 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.952948 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.952984 17706 log.cc:1199] T bbdc98a938b247bdaf113dc94c3b4d59 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/bbdc98a938b247bdaf113dc94c3b4d59
I20260929 02:01:08.953526 17706 ts_tablet_manager.cc:1996] T bbdc98a938b247bdaf113dc94c3b4d59 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:08.955066 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.955430 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.955704 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.955996 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:08.957880 17708 ts_tablet_manager.cc:1975] T 5acedab1b27c414fa5181e35aa9ac24b P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.958046 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.958213 17708 log.cc:1199] T 5acedab1b27c414fa5181e35aa9ac24b P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/5acedab1b27c414fa5181e35aa9ac24b
I20260929 02:01:08.958382 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.958645 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.959074 17708 ts_tablet_manager.cc:1996] T 5acedab1b27c414fa5181e35aa9ac24b P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:08.960353 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 5acedab1b27c414fa5181e35aa9ac24b (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:08.961091 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:08.961527 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.961690 17708 tablet_replica.cc:333] T c6cac2e61f6f4795869af850549096a9 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:08.961956 17708 raft_consensus.cc:2243] T c6cac2e61f6f4795869af850549096a9 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:08.962407 17707 ts_tablet_manager.cc:1975] T ae7f6651c994403697a0f04508b70dae P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.962606 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.003s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":140,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:08.962733 17708 raft_consensus.cc:2272] T c6cac2e61f6f4795869af850549096a9 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.962985 17707 log.cc:1199] T ae7f6651c994403697a0f04508b70dae P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/ae7f6651c994403697a0f04508b70dae
I20260929 02:01:08.963126 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.154528
I20260929 02:01:08.963506 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.154528
I20260929 02:01:08.963816 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.154528
I20260929 02:01:08.964195 17708 ts_tablet_manager.cc:1962] T c6cac2e61f6f4795869af850549096a9 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.965234 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.016s user 0.001s sys 0.004s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":138,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:08.967978 17707 ts_tablet_manager.cc:1996] T ae7f6651c994403697a0f04508b70dae P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:08.968652 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.968889 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.969257 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet ae7f6651c994403697a0f04508b70dae (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:08.969550 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:08.970002 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:08.970119 17711 tablet_bootstrap.cc:492] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:08.970297 17711 ts_tablet_manager.cc:1449] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.053s user 0.008s sys 0.020s
I20260929 02:01:08.970821 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.018s user 0.005s sys 0.003s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5401,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":170,"lbm_read_time_us":554,"lbm_reads_lt_1ms":27,"lbm_write_time_us":307,"lbm_writes_lt_1ms":9,"mutex_wait_us":4135,"peak_mem_usage":32768,"reinsert_count":0,"update_count":2}
I20260929 02:01:08.971146 17711 raft_consensus.cc:359] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.971417 17711 raft_consensus.cc:385] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:08.971511 17711 raft_consensus.cc:740] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:08.971820 17711 consensus_queue.cc:260] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.972482 17711 ts_tablet_manager.cc:1480] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 02:01:08.972863 17711 tablet_bootstrap.cc:492] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:08.974781 17711 tablet_bootstrap.cc:654] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:08.978116 17707 tablet_replica.cc:333] T 8f328eef30d541c7a61e7d7403e597c7 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:08.978585 17707 raft_consensus.cc:2243] T 8f328eef30d541c7a61e7d7403e597c7 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:08.978832 17707 raft_consensus.cc:2272] T 8f328eef30d541c7a61e7d7403e597c7 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.980266 17707 ts_tablet_manager.cc:1962] T 8f328eef30d541c7a61e7d7403e597c7 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.981055 17594 tablet_bootstrap.cc:492] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:08.981230 17594 ts_tablet_manager.cc:1449] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.080s user 0.002s sys 0.005s
I20260929 02:01:08.981990 17594 raft_consensus.cc:359] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.982197 17594 raft_consensus.cc:385] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:08.982277 17594 raft_consensus.cc:740] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:08.982537 17594 consensus_queue.cc:260] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.983139 17594 ts_tablet_manager.cc:1480] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 02:01:08.983585 17594 tablet_bootstrap.cc:492] T 5d8486054d464e4ab45c10da4de28090 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:08.985541 17594 tablet_bootstrap.cc:654] T 5d8486054d464e4ab45c10da4de28090 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:08.986500 17711 tablet_bootstrap.cc:492] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:08.986709 17711 ts_tablet_manager.cc:1449] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.014s user 0.001s sys 0.002s
I20260929 02:01:08.987563 17711 raft_consensus.cc:359] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.988703 17711 raft_consensus.cc:385] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:08.988524 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.023s user 0.010s sys 0.002s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5401,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":2418,"dirs.run_cpu_time_us":281,"dirs.run_wall_time_us":6389,"lbm_read_time_us":546,"lbm_reads_lt_1ms":27,"lbm_write_time_us":403,"lbm_writes_lt_1ms":9,"peak_mem_usage":32768,"reinsert_count":0,"update_count":2}
I20260929 02:01:08.988835 17711 raft_consensus.cc:740] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:08.988605 17708 ts_tablet_manager.cc:1975] T c6cac2e61f6f4795869af850549096a9 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:08.989079 17708 log.cc:1199] T c6cac2e61f6f4795869af850549096a9 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/c6cac2e61f6f4795869af850549096a9
I20260929 02:01:08.989518 17708 ts_tablet_manager.cc:1996] T c6cac2e61f6f4795869af850549096a9 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:08.989686 17543 raft_consensus.cc:493] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:08.989918 17543 raft_consensus.cc:515] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.989732 17711 consensus_queue.cc:260] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:08.990840 17543 leader_election.cc:290] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:08.991377 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet c6cac2e61f6f4795869af850549096a9 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:08.992090 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c029d8ad8e534719be68303011ec7d8b" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:08.992368 16984 raft_consensus.cc:2468] T c029d8ad8e534719be68303011ec7d8b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:08.992798 17711 ts_tablet_manager.cc:1480] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.006s user 0.001s sys 0.001s
I20260929 02:01:08.993521 17711 tablet_bootstrap.cc:492] T 16931d61d19f4b1eaed27200e43c5336 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:08.993484 17056 leader_election.cc:304] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:08.995213 17708 tablet_replica.cc:333] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:08.995556 17708 raft_consensus.cc:2243] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:08.995954 17708 raft_consensus.cc:2272] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:08.996316 17543 raft_consensus.cc:2804] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:08.996424 17543 raft_consensus.cc:493] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:08.996608 17543 raft_consensus.cc:3060] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:08.997018 17708 ts_tablet_manager.cc:1962] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:08.999056 17711 tablet_bootstrap.cc:654] T 16931d61d19f4b1eaed27200e43c5336 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:09.000981 17594 tablet_bootstrap.cc:492] T 5d8486054d464e4ab45c10da4de28090 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:09.001150 17594 ts_tablet_manager.cc:1449] T 5d8486054d464e4ab45c10da4de28090 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.018s user 0.002s sys 0.007s
I20260929 02:01:09.002005 17594 raft_consensus.cc:359] T 5d8486054d464e4ab45c10da4de28090 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.002200 17594 raft_consensus.cc:385] T 5d8486054d464e4ab45c10da4de28090 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.002303 17594 raft_consensus.cc:740] T 5d8486054d464e4ab45c10da4de28090 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.002580 17594 consensus_queue.cc:260] T 5d8486054d464e4ab45c10da4de28090 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.005422 17708 ts_tablet_manager.cc:1975] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:09.005568 17708 log.cc:1199] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/ae7f6651c994403697a0f04508b70dae
I20260929 02:01:09.005935 17708 ts_tablet_manager.cc:1996] T ae7f6651c994403697a0f04508b70dae P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:09.006340 17594 ts_tablet_manager.cc:1480] T 5d8486054d464e4ab45c10da4de28090 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260929 02:01:09.006835 17708 tablet_replica.cc:333] T c7892f83a6e34ba0899bb3ac41a749a5 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:09.008220 17625 raft_consensus.cc:493] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:09.008493 17625 raft_consensus.cc:515] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.008783 17594 tablet_bootstrap.cc:492] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:09.008908 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet ae7f6651c994403697a0f04508b70dae (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:09.009248 17625 leader_election.cc:290] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:09.010767 17594 tablet_bootstrap.cc:654] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:09.012323 17707 ts_tablet_manager.cc:1975] T 8f328eef30d541c7a61e7d7403e597c7 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:09.012248 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5aa0d7aa540479c89ae9b8b61d89c4e" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:09.012534 17118 raft_consensus.cc:2468] T e5aa0d7aa540479c89ae9b8b61d89c4e P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:09.013043 17708 raft_consensus.cc:2243] T c7892f83a6e34ba0899bb3ac41a749a5 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:09.013165 17707 log.cc:1199] T 8f328eef30d541c7a61e7d7403e597c7 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/8f328eef30d541c7a61e7d7403e597c7
I20260929 02:01:09.013278 17708 raft_consensus.cc:2272] T c7892f83a6e34ba0899bb3ac41a749a5 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:09.013950 17707 ts_tablet_manager.cc:1996] T 8f328eef30d541c7a61e7d7403e597c7 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:09.015137 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 8f328eef30d541c7a61e7d7403e597c7 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
W20260929 02:01:09.016057 16800 catalog_manager.cc:5030] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): delete failed for tablet a21bdeb06bb24a45945e00b68b17ca5b because the tablet was not found. No further retry: Not found: Tablet not found: a21bdeb06bb24a45945e00b68b17ca5b
I20260929 02:01:09.016108 17707 tablet_replica.cc:333] T b402b4aee2a44330a87657f5a9469dc0 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:09.016350 17707 raft_consensus.cc:2243] T b402b4aee2a44330a87657f5a9469dc0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:09.016575 17707 raft_consensus.cc:2272] T b402b4aee2a44330a87657f5a9469dc0 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:09.016521 17543 raft_consensus.cc:515] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.016788 16919 leader_election.cc:304] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:09.017319 17337 raft_consensus.cc:2804] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:09.017441 17337 raft_consensus.cc:493] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:09.017592 17337 raft_consensus.cc:3060] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.017822 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c029d8ad8e534719be68303011ec7d8b" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:09.018049 16984 raft_consensus.cc:3060] T c029d8ad8e534719be68303011ec7d8b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.020771 17708 ts_tablet_manager.cc:1962] T c7892f83a6e34ba0899bb3ac41a749a5 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:09.020787 17625 raft_consensus.cc:493] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:09.021399 17543 leader_election.cc:290] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:09.021565 17625 raft_consensus.cc:515] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.021790 17337 raft_consensus.cc:515] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.022071 16984 raft_consensus.cc:2468] T c029d8ad8e534719be68303011ec7d8b P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:09.022188 17625 leader_election.cc:290] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:08.962072 17595 tablet_bootstrap.cc:492] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:09.022622 17595 ts_tablet_manager.cc:1449] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.092s user 0.001s sys 0.007s
I20260929 02:01:09.022959 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d8486054d464e4ab45c10da4de28090" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:09.023180 17118 raft_consensus.cc:2468] T 5d8486054d464e4ab45c10da4de28090 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:09.023553 17252 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d8486054d464e4ab45c10da4de28090" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:09.023727 17711 tablet_bootstrap.cc:492] T 16931d61d19f4b1eaed27200e43c5336 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:09.023654 17595 raft_consensus.cc:359] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.023860 17595 raft_consensus.cc:385] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.023833 17252 raft_consensus.cc:2468] T 5d8486054d464e4ab45c10da4de28090 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:09.023185 17056 leader_election.cc:304] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:09.023941 17595 raft_consensus.cc:740] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.024220 17595 consensus_queue.cc:260] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.024837 17595 ts_tablet_manager.cc:1480] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 02:01:08.994124 17250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c029d8ad8e534719be68303011ec7d8b" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:09.025058 17711 ts_tablet_manager.cc:1449] T 16931d61d19f4b1eaed27200e43c5336 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.031s user 0.001s sys 0.002s
I20260929 02:01:09.009991 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5aa0d7aa540479c89ae9b8b61d89c4e" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:09.017863 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c029d8ad8e534719be68303011ec7d8b" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
W20260929 02:01:09.025653 17055 leader_election.cc:343] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:09.025694 17707 ts_tablet_manager.cc:1962] T b402b4aee2a44330a87657f5a9469dc0 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 02:01:09.026057 17055 leader_election.cc:343] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:09.026328 17711 raft_consensus.cc:359] T 16931d61d19f4b1eaed27200e43c5336 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.026996 16919 leader_election.cc:304] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:09.027275 17595 tablet_bootstrap.cc:492] T c029d8ad8e534719be68303011ec7d8b P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
W20260929 02:01:09.027875 16921 leader_election.cc:343] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:09.028501 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5aa0d7aa540479c89ae9b8b61d89c4e" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:09.028676 17118 raft_consensus.cc:3060] T e5aa0d7aa540479c89ae9b8b61d89c4e P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.029186 17595 tablet_bootstrap.cc:654] T c029d8ad8e534719be68303011ec7d8b P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:09.030249 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5aa0d7aa540479c89ae9b8b61d89c4e" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:09.031584 17118 raft_consensus.cc:2468] T e5aa0d7aa540479c89ae9b8b61d89c4e P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:09.031648 17543 raft_consensus.cc:2804] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:09.032526 17543 raft_consensus.cc:697] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Becoming Leader. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Running, Role: LEADER
I20260929 02:01:09.032882 17543 consensus_queue.cc:237] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.035158 17337 leader_election.cc:290] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Requested vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:09.036103 17711 raft_consensus.cc:385] T 16931d61d19f4b1eaed27200e43c5336 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.036222 17711 raft_consensus.cc:740] T 16931d61d19f4b1eaed27200e43c5336 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.036509 17711 consensus_queue.cc:260] T 16931d61d19f4b1eaed27200e43c5336 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.037014 17711 ts_tablet_manager.cc:1480] T 16931d61d19f4b1eaed27200e43c5336 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.012s user 0.000s sys 0.002s
I20260929 02:01:09.037340 17711 tablet_bootstrap.cc:492] T 800999ad9c144b02b58cf1bf02b6d9e2 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:09.038532 17625 raft_consensus.cc:2804] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:09.039021 17625 raft_consensus.cc:493] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
W20260929 02:01:09.039048 16921 leader_election.cc:343] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:09.039510 16919 leader_election.cc:304] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters: d103dd4823044faebe46100a0dfbee88
I20260929 02:01:09.039969 17624 raft_consensus.cc:2804] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:09.039144 17625 raft_consensus.cc:3060] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.040103 17624 raft_consensus.cc:697] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Becoming Leader. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Running, Role: LEADER
I20260929 02:01:09.040354 17706 tablet_replica.cc:333] T 7239b23c6e30483d9bd0738e07c63610 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:09.040377 17624 consensus_queue.cc:237] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.040611 17706 raft_consensus.cc:2243] T 7239b23c6e30483d9bd0738e07c63610 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:09.040799 17706 raft_consensus.cc:2272] T 7239b23c6e30483d9bd0738e07c63610 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:09.041676 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet bbdc98a938b247bdaf113dc94c3b4d59 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:09.043634 17625 raft_consensus.cc:515] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.045419 17706 ts_tablet_manager.cc:1962] T 7239b23c6e30483d9bd0738e07c63610 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:09.045728 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d8486054d464e4ab45c10da4de28090" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:09.045982 17625 leader_election.cc:290] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Requested vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:09.046015 17253 raft_consensus.cc:3060] T 5d8486054d464e4ab45c10da4de28090 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.046990 17711 tablet_bootstrap.cc:654] T 800999ad9c144b02b58cf1bf02b6d9e2 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:09.047730 17708 ts_tablet_manager.cc:1975] T c7892f83a6e34ba0899bb3ac41a749a5 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:09.047936 17708 log.cc:1199] T c7892f83a6e34ba0899bb3ac41a749a5 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/c7892f83a6e34ba0899bb3ac41a749a5
I20260929 02:01:09.048563 17708 ts_tablet_manager.cc:1996] T c7892f83a6e34ba0899bb3ac41a749a5 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:09.048810 17595 tablet_bootstrap.cc:492] T c029d8ad8e534719be68303011ec7d8b P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:09.048985 17595 ts_tablet_manager.cc:1449] T c029d8ad8e534719be68303011ec7d8b P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.022s user 0.001s sys 0.005s
I20260929 02:01:09.050014 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet c7892f83a6e34ba0899bb3ac41a749a5 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:09.050086 17708 tablet_replica.cc:333] T 306f1984e9324a949db3435c56fcfb8f P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:09.050082 17253 raft_consensus.cc:2468] T 5d8486054d464e4ab45c10da4de28090 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:09.050026 17595 raft_consensus.cc:359] T c029d8ad8e534719be68303011ec7d8b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.050415 17595 raft_consensus.cc:385] T c029d8ad8e534719be68303011ec7d8b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.050462 17708 raft_consensus.cc:2243] T 306f1984e9324a949db3435c56fcfb8f P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:09.050535 17595 raft_consensus.cc:740] T c029d8ad8e534719be68303011ec7d8b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.050580 16921 leader_election.cc:304] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:09.050868 17708 raft_consensus.cc:2272] T 306f1984e9324a949db3435c56fcfb8f P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:09.050799 17595 consensus_queue.cc:260] T c029d8ad8e534719be68303011ec7d8b P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.052302 17595 ts_tablet_manager.cc:1480] T c029d8ad8e534719be68303011ec7d8b P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.003s user 0.003s sys 0.000s
I20260929 02:01:09.052708 17595 tablet_bootstrap.cc:492] T 26ea597431574ea9a145a78687442f50 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:09.051004 17625 raft_consensus.cc:2804] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:09.052435 17708 ts_tablet_manager.cc:1962] T 306f1984e9324a949db3435c56fcfb8f P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:09.057014 17625 raft_consensus.cc:697] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Becoming Leader. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Running, Role: LEADER
I20260929 02:01:09.057324 17625 consensus_queue.cc:237] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.059509 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5d8486054d464e4ab45c10da4de28090" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:09.062631 17118 raft_consensus.cc:3060] T 5d8486054d464e4ab45c10da4de28090 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.064890 16819 catalog_manager.cc:5838] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97df55a0361841319adfae27c9787b2f (127.0.71.65). New cstate: current_term: 1 leader_uuid: "97df55a0361841319adfae27c9787b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:09.066138 17118 raft_consensus.cc:2468] T 5d8486054d464e4ab45c10da4de28090 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:09.068302 17711 tablet_bootstrap.cc:492] T 800999ad9c144b02b58cf1bf02b6d9e2 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:09.068317 17595 tablet_bootstrap.cc:654] T 26ea597431574ea9a145a78687442f50 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:09.069476 17711 ts_tablet_manager.cc:1449] T 800999ad9c144b02b58cf1bf02b6d9e2 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.032s user 0.007s sys 0.000s
I20260929 02:01:09.071070 17711 raft_consensus.cc:359] T 800999ad9c144b02b58cf1bf02b6d9e2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.071415 17711 raft_consensus.cc:385] T 800999ad9c144b02b58cf1bf02b6d9e2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.071580 17711 raft_consensus.cc:740] T 800999ad9c144b02b58cf1bf02b6d9e2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.071614 16818 catalog_manager.cc:5838] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66). New cstate: current_term: 1 leader_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:09.072072 17711 consensus_queue.cc:260] T 800999ad9c144b02b58cf1bf02b6d9e2 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.072997 17711 ts_tablet_manager.cc:1480] T 800999ad9c144b02b58cf1bf02b6d9e2 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260929 02:01:09.073391 17711 tablet_bootstrap.cc:492] T 273fd95488284ef88c2272052f0de976 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:09.080199 16817 catalog_manager.cc:5838] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97df55a0361841319adfae27c9787b2f (127.0.71.65). New cstate: current_term: 1 leader_uuid: "97df55a0361841319adfae27c9787b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:09.080638 17706 ts_tablet_manager.cc:1975] T 7239b23c6e30483d9bd0738e07c63610 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:09.080803 17706 log.cc:1199] T 7239b23c6e30483d9bd0738e07c63610 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/7239b23c6e30483d9bd0738e07c63610
I20260929 02:01:09.081295 17706 ts_tablet_manager.cc:1996] T 7239b23c6e30483d9bd0738e07c63610 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:09.082434 17706 tablet_replica.cc:333] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:09.082646 17706 raft_consensus.cc:2243] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:09.083281 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.082948 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 7239b23c6e30483d9bd0738e07c63610 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:09.087809 17625 raft_consensus.cc:493] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:09.087976 17625 raft_consensus.cc:515] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.087926 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.088685 17625 leader_election.cc:290] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:09.090077 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b094aa976542c2bf9646de8377fe18" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:09.090353 17253 raft_consensus.cc:2468] T b8b094aa976542c2bf9646de8377fe18 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:09.090646 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b094aa976542c2bf9646de8377fe18" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:09.090955 16921 leader_election.cc:304] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:09.091352 17625 raft_consensus.cc:2804] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:09.091478 17625 raft_consensus.cc:493] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:09.091354 17118 raft_consensus.cc:2468] T b8b094aa976542c2bf9646de8377fe18 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:09.091579 17625 raft_consensus.cc:3060] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.093395 17706 raft_consensus.cc:2272] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:09.095425 17707 ts_tablet_manager.cc:1975] T b402b4aee2a44330a87657f5a9469dc0 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:09.095655 17707 log.cc:1199] T b402b4aee2a44330a87657f5a9469dc0 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/b402b4aee2a44330a87657f5a9469dc0
I20260929 02:01:09.096256 17707 ts_tablet_manager.cc:1996] T b402b4aee2a44330a87657f5a9469dc0 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:09.096724 17706 ts_tablet_manager.cc:1962] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:09.098310 17625 raft_consensus.cc:515] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.099398 17708 ts_tablet_manager.cc:1975] T 306f1984e9324a949db3435c56fcfb8f P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:09.099547 17708 log.cc:1199] T 306f1984e9324a949db3435c56fcfb8f P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/306f1984e9324a949db3435c56fcfb8f
I20260929 02:01:09.099778 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.100085 17708 ts_tablet_manager.cc:1996] T 306f1984e9324a949db3435c56fcfb8f P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:09.100162 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.100546 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.101598 17594 tablet_bootstrap.cc:492] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:09.101824 17594 ts_tablet_manager.cc:1449] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.093s user 0.006s sys 0.005s
I20260929 02:01:09.102061 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b094aa976542c2bf9646de8377fe18" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:09.102319 17253 raft_consensus.cc:3060] T b8b094aa976542c2bf9646de8377fe18 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.102790 17594 raft_consensus.cc:359] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.103015 17594 raft_consensus.cc:385] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.103086 17594 raft_consensus.cc:740] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.103410 17594 consensus_queue.cc:260] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.103996 17594 ts_tablet_manager.cc:1480] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 02:01:09.075644 17711 tablet_bootstrap.cc:654] T 273fd95488284ef88c2272052f0de976 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:09.104646 17708 tablet_replica.cc:333] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:09.104894 17708 raft_consensus.cc:2243] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:09.105334 17708 raft_consensus.cc:2272] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:09.106467 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.107483 17708 ts_tablet_manager.cc:1962] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:09.109378 17625 leader_election.cc:290] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Requested vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:09.109622 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.110066 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.110868 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 306f1984e9324a949db3435c56fcfb8f (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:09.111361 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b8b094aa976542c2bf9646de8377fe18" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:09.112805 17118 raft_consensus.cc:3060] T b8b094aa976542c2bf9646de8377fe18 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.112635 17594 tablet_bootstrap.cc:492] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:09.114560 17595 tablet_bootstrap.cc:492] T 26ea597431574ea9a145a78687442f50 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:09.114753 17595 ts_tablet_manager.cc:1449] T 26ea597431574ea9a145a78687442f50 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.062s user 0.015s sys 0.015s
I20260929 02:01:09.115550 17595 raft_consensus.cc:359] T 26ea597431574ea9a145a78687442f50 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.115748 17595 raft_consensus.cc:385] T 26ea597431574ea9a145a78687442f50 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.115814 17595 raft_consensus.cc:740] T 26ea597431574ea9a145a78687442f50 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.116112 17595 consensus_queue.cc:260] T 26ea597431574ea9a145a78687442f50 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.116384 17707 tablet_replica.cc:333] T 2e0146fb337d43b2921fb5fd34fb95ed P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:09.116659 17707 raft_consensus.cc:2243] T 2e0146fb337d43b2921fb5fd34fb95ed P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:09.116566 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet b402b4aee2a44330a87657f5a9469dc0 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:09.116869 17707 raft_consensus.cc:2272] T 2e0146fb337d43b2921fb5fd34fb95ed P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:09.119344 17707 ts_tablet_manager.cc:1962] T 2e0146fb337d43b2921fb5fd34fb95ed P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:09.120209 17594 tablet_bootstrap.cc:654] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:09.120266 17595 ts_tablet_manager.cc:1480] T 26ea597431574ea9a145a78687442f50 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.005s user 0.001s sys 0.001s
I20260929 02:01:09.121294 17682 raft_consensus.cc:493] T 26ea597431574ea9a145a78687442f50 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:09.121447 17682 raft_consensus.cc:515] T 26ea597431574ea9a145a78687442f50 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.121893 17118 raft_consensus.cc:2468] T b8b094aa976542c2bf9646de8377fe18 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:09.122047 17682 leader_election.cc:290] T 26ea597431574ea9a145a78687442f50 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:09.122737 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26ea597431574ea9a145a78687442f50" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:09.123235 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26ea597431574ea9a145a78687442f50" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:09.122694 16919 leader_election.cc:304] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
W20260929 02:01:09.123684 17186 leader_election.cc:343] T 26ea597431574ea9a145a78687442f50 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
W20260929 02:01:09.123926 17189 leader_election.cc:343] T 26ea597431574ea9a145a78687442f50 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:09.124037 17189 leader_election.cc:304] T 26ea597431574ea9a145a78687442f50 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f
I20260929 02:01:09.124413 17682 raft_consensus.cc:2749] T 26ea597431574ea9a145a78687442f50 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:09.124966 17595 tablet_bootstrap.cc:492] T fed81261ad7d4afabab4ada3e33aa5e2 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:09.126761 17595 tablet_bootstrap.cc:654] T fed81261ad7d4afabab4ada3e33aa5e2 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:09.127523 17625 raft_consensus.cc:2804] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:09.127797 17625 raft_consensus.cc:697] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Becoming Leader. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Running, Role: LEADER
I20260929 02:01:09.128108 17625 consensus_queue.cc:237] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.128830 17707 ts_tablet_manager.cc:1975] T 2e0146fb337d43b2921fb5fd34fb95ed P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:09.128952 17707 log.cc:1199] T 2e0146fb337d43b2921fb5fd34fb95ed P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/2e0146fb337d43b2921fb5fd34fb95ed
I20260929 02:01:09.129429 17707 ts_tablet_manager.cc:1996] T 2e0146fb337d43b2921fb5fd34fb95ed P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:09.130995 17706 ts_tablet_manager.cc:1975] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:09.131138 17706 log.cc:1199] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/8f328eef30d541c7a61e7d7403e597c7
I20260929 02:01:09.133466 17706 ts_tablet_manager.cc:1996] T 8f328eef30d541c7a61e7d7403e597c7 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:09.134614 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 8f328eef30d541c7a61e7d7403e597c7 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
W20260929 02:01:09.135295 16802 catalog_manager.cc:5030] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): delete failed for tablet a21bdeb06bb24a45945e00b68b17ca5b because the tablet was not found. No further retry: Not found: Tablet not found: a21bdeb06bb24a45945e00b68b17ca5b
I20260929 02:01:09.135579 17706 tablet_replica.cc:333] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:09.135795 17706 raft_consensus.cc:2243] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:09.136108 17706 raft_consensus.cc:2272] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:09.137043 17706 ts_tablet_manager.cc:1962] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:09.137811 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 2e0146fb337d43b2921fb5fd34fb95ed (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:09.138538 17594 tablet_bootstrap.cc:492] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:09.138700 17594 ts_tablet_manager.cc:1449] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.026s user 0.005s sys 0.001s
I20260929 02:01:09.139482 17707 tablet_replica.cc:333] T 05f61c0da0dd4a20a60cc2b3390cb07c P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:09.139729 17707 raft_consensus.cc:2243] T 05f61c0da0dd4a20a60cc2b3390cb07c P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:09.139911 17707 raft_consensus.cc:2272] T 05f61c0da0dd4a20a60cc2b3390cb07c P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:09.139590 17594 raft_consensus.cc:359] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.140084 17594 raft_consensus.cc:385] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.140169 17594 raft_consensus.cc:740] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.140506 17594 consensus_queue.cc:260] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.141932 17594 ts_tablet_manager.cc:1480] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:01:09.142256 17594 tablet_bootstrap.cc:492] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:09.146811 17707 ts_tablet_manager.cc:1962] T 05f61c0da0dd4a20a60cc2b3390cb07c P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:09.150746 17708 ts_tablet_manager.cc:1975] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:09.150889 17708 log.cc:1199] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/bbdc98a938b247bdaf113dc94c3b4d59
I20260929 02:01:09.151350 17708 ts_tablet_manager.cc:1996] T bbdc98a938b247bdaf113dc94c3b4d59 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:09.152556 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet bbdc98a938b247bdaf113dc94c3b4d59 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:09.153497 17594 tablet_bootstrap.cc:654] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:09.156582 17595 tablet_bootstrap.cc:492] T fed81261ad7d4afabab4ada3e33aa5e2 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:09.156774 17595 ts_tablet_manager.cc:1449] T fed81261ad7d4afabab4ada3e33aa5e2 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.032s user 0.003s sys 0.011s
I20260929 02:01:09.158128 17595 raft_consensus.cc:359] T fed81261ad7d4afabab4ada3e33aa5e2 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.158325 17595 raft_consensus.cc:385] T fed81261ad7d4afabab4ada3e33aa5e2 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.158432 17595 raft_consensus.cc:740] T fed81261ad7d4afabab4ada3e33aa5e2 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.160216 17595 consensus_queue.cc:260] T fed81261ad7d4afabab4ada3e33aa5e2 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.158212 17706 ts_tablet_manager.cc:1975] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:09.160705 17706 log.cc:1199] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/2e0146fb337d43b2921fb5fd34fb95ed
I20260929 02:01:09.161181 17706 ts_tablet_manager.cc:1996] T 2e0146fb337d43b2921fb5fd34fb95ed P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:09.163718 17595 ts_tablet_manager.cc:1480] T fed81261ad7d4afabab4ada3e33aa5e2 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.007s user 0.002s sys 0.000s
I20260929 02:01:09.166129 17706 tablet_replica.cc:333] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:09.166478 17706 raft_consensus.cc:2243] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:09.166651 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 2e0146fb337d43b2921fb5fd34fb95ed (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:09.167541 17707 ts_tablet_manager.cc:1975] T 05f61c0da0dd4a20a60cc2b3390cb07c P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:09.167866 17595 tablet_bootstrap.cc:492] T 964849b43d28439aa45e8a51084bcae9 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:09.168256 17706 raft_consensus.cc:2272] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:09.168843 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.080s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":224,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:09.169224 17708 tablet_replica.cc:333] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:09.169353 17706 ts_tablet_manager.cc:1962] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:09.172061 17595 tablet_bootstrap.cc:654] T 964849b43d28439aa45e8a51084bcae9 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:09.172535 17708 raft_consensus.cc:2243] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:09.172811 17707 log.cc:1199] T 05f61c0da0dd4a20a60cc2b3390cb07c P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/05f61c0da0dd4a20a60cc2b3390cb07c
I20260929 02:01:09.172989 17708 raft_consensus.cc:2272] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:09.173282 17707 ts_tablet_manager.cc:1996] T 05f61c0da0dd4a20a60cc2b3390cb07c P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:09.173873 17543 raft_consensus.cc:493] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:09.173931 17708 ts_tablet_manager.cc:1962] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:09.174033 17543 raft_consensus.cc:515] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.174823 17543 leader_election.cc:290] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:09.175546 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fed81261ad7d4afabab4ada3e33aa5e2" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:09.175736 17251 raft_consensus.cc:2468] T fed81261ad7d4afabab4ada3e33aa5e2 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:09.176558 17055 leader_election.cc:304] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:09.176695 17711 tablet_bootstrap.cc:492] T 273fd95488284ef88c2272052f0de976 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:09.176888 17711 ts_tablet_manager.cc:1449] T 273fd95488284ef88c2272052f0de976 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.104s user 0.001s sys 0.025s
I20260929 02:01:09.177030 17543 raft_consensus.cc:2804] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:09.177187 17543 raft_consensus.cc:493] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:09.177300 17543 raft_consensus.cc:3060] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.177254 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fed81261ad7d4afabab4ada3e33aa5e2" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:09.177461 16984 raft_consensus.cc:2468] T fed81261ad7d4afabab4ada3e33aa5e2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:09.177834 17711 raft_consensus.cc:359] T 273fd95488284ef88c2272052f0de976 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.178004 17711 raft_consensus.cc:385] T 273fd95488284ef88c2272052f0de976 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.178074 17711 raft_consensus.cc:740] T 273fd95488284ef88c2272052f0de976 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.178318 17711 consensus_queue.cc:260] T 273fd95488284ef88c2272052f0de976 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.178859 17711 ts_tablet_manager.cc:1480] T 273fd95488284ef88c2272052f0de976 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.002s user 0.000s sys 0.001s
I20260929 02:01:09.179133 17711 tablet_bootstrap.cc:492] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:09.180160 17543 raft_consensus.cc:515] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.180872 17711 tablet_bootstrap.cc:654] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:09.181360 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fed81261ad7d4afabab4ada3e33aa5e2" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:09.181564 16984 raft_consensus.cc:3060] T fed81261ad7d4afabab4ada3e33aa5e2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.184628 16984 raft_consensus.cc:2468] T fed81261ad7d4afabab4ada3e33aa5e2 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:09.185129 17056 leader_election.cc:304] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:09.185412 17398 raft_consensus.cc:2804] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:09.185568 17398 raft_consensus.cc:697] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Becoming Leader. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Running, Role: LEADER
I20260929 02:01:09.185840 17398 consensus_queue.cc:237] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.188351 17543 leader_election.cc:290] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:09.188591 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.078s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":78196}
I20260929 02:01:09.188946 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.079s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":78492}
I20260929 02:01:09.190260 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.157775
I20260929 02:01:09.190595 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.157775
I20260929 02:01:09.190845 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.157775
I20260929 02:01:09.191490 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "fed81261ad7d4afabab4ada3e33aa5e2" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:09.191524 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.191690 17251 raft_consensus.cc:3060] T fed81261ad7d4afabab4ada3e33aa5e2 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.191784 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.193958 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:09.194205 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.197909 17682 raft_consensus.cc:493] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:09.197721 16818 catalog_manager.cc:5838] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66). New cstate: current_term: 1 leader_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:09.198086 17682 raft_consensus.cc:515] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.198765 17682 leader_election.cc:290] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:09.199745 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8c4b957e16467fa6bd6e9c317b1b1f" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:09.199968 17118 raft_consensus.cc:2468] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 0.
I20260929 02:01:09.200510 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8c4b957e16467fa6bd6e9c317b1b1f" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:09.200659 17186 leader_election.cc:304] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters:
W20260929 02:01:09.201071 17189 leader_election.cc:343] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:09.201160 17251 raft_consensus.cc:2468] T fed81261ad7d4afabab4ada3e33aa5e2 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:09.201753 17682 raft_consensus.cc:2804] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:09.201855 17682 raft_consensus.cc:493] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:09.201969 17682 raft_consensus.cc:3060] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.202435 17594 tablet_bootstrap.cc:492] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:09.202562 17708 ts_tablet_manager.cc:1975] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:09.202600 17594 ts_tablet_manager.cc:1449] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.060s user 0.005s sys 0.000s
I20260929 02:01:09.202703 17708 log.cc:1199] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/7239b23c6e30483d9bd0738e07c63610
I20260929 02:01:09.203169 17708 ts_tablet_manager.cc:1996] T 7239b23c6e30483d9bd0738e07c63610 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:09.203540 17594 raft_consensus.cc:359] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.203739 17594 raft_consensus.cc:385] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.203811 17594 raft_consensus.cc:740] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.204255 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 7239b23c6e30483d9bd0738e07c63610 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:09.204022 17594 consensus_queue.cc:260] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.205057 17594 ts_tablet_manager.cc:1480] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:01:09.205976 17594 tablet_bootstrap.cc:492] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:09.206300 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.099s user 0.002s sys 0.004s Metrics: {"bytes_written":466,"delete_count":0,"lbm_write_time_us":152,"lbm_writes_lt_1ms":4,"mutex_wait_us":67816,"reinsert_count":0,"update_count":2}
I20260929 02:01:09.207865 17674 raft_consensus.cc:493] T c029d8ad8e534719be68303011ec7d8b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:09.208014 17674 raft_consensus.cc:515] T c029d8ad8e534719be68303011ec7d8b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.208699 17674 leader_election.cc:290] T c029d8ad8e534719be68303011ec7d8b P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:09.210239 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c029d8ad8e534719be68303011ec7d8b" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:09.210306 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "c029d8ad8e534719be68303011ec7d8b" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:09.210547 16984 raft_consensus.cc:2393] T c029d8ad8e534719be68303011ec7d8b P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d103dd4823044faebe46100a0dfbee88 in current term 1: Already voted for candidate 8668afa5b5ca4ebdabc812d0789de7bc in this term.
I20260929 02:01:09.211126 17186 leader_election.cc:304] T c029d8ad8e534719be68303011ec7d8b P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f
I20260929 02:01:09.211829 17674 raft_consensus.cc:3060] T c029d8ad8e534719be68303011ec7d8b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.212662 17708 tablet_replica.cc:333] T 8f328eef30d541c7a61e7d7403e597c7 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:09.212903 17708 raft_consensus.cc:2243] T 8f328eef30d541c7a61e7d7403e597c7 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:09.213505 17708 raft_consensus.cc:2272] T 8f328eef30d541c7a61e7d7403e597c7 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:09.214152 17708 ts_tablet_manager.cc:1962] T 8f328eef30d541c7a61e7d7403e597c7 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:09.214919 17594 tablet_bootstrap.cc:654] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:09.215355 17674 raft_consensus.cc:2749] T c029d8ad8e534719be68303011ec7d8b P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:09.216305 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.205782 17682 raft_consensus.cc:515] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.217576 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.218202 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8c4b957e16467fa6bd6e9c317b1b1f" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:09.218370 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8c4b957e16467fa6bd6e9c317b1b1f" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:09.218549 17118 raft_consensus.cc:3060] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.219200 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.118s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":118111}
I20260929 02:01:09.219564 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.119s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":158,"lbm_writes_lt_1ms":4,"mutex_wait_us":108643,"reinsert_count":0,"update_count":1}
I20260929 02:01:09.220055 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.234039
I20260929 02:01:09.220336 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.234039
I20260929 02:01:09.220407 17706 ts_tablet_manager.cc:1975] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:09.220538 17706 log.cc:1199] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/b402b4aee2a44330a87657f5a9469dc0
W20260929 02:01:09.220763 17189 leader_election.cc:343] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:09.220990 17706 ts_tablet_manager.cc:1996] T b402b4aee2a44330a87657f5a9469dc0 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:09.221864 17706 tablet_replica.cc:333] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:09.221908 17118 raft_consensus.cc:2468] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 1.
I20260929 02:01:09.222059 17706 raft_consensus.cc:2243] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:09.222350 17706 raft_consensus.cc:2272] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:09.222509 17186 leader_election.cc:304] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters: 97df55a0361841319adfae27c9787b2f
I20260929 02:01:09.222970 17674 raft_consensus.cc:2804] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:09.223034 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet b402b4aee2a44330a87657f5a9469dc0 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:09.223620 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.224612 17674 raft_consensus.cc:697] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Becoming Leader. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Running, Role: LEADER
I20260929 02:01:09.225278 17674 consensus_queue.cc:237] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.224807 17682 leader_election.cc:290] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:09.225987 17706 ts_tablet_manager.cc:1962] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:09.228645 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 05f61c0da0dd4a20a60cc2b3390cb07c (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:09.229979 17711 tablet_bootstrap.cc:492] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:09.233876 17711 ts_tablet_manager.cc:1449] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.055s user 0.005s sys 0.005s
I20260929 02:01:09.233927 17253 raft_consensus.cc:2468] T b8b094aa976542c2bf9646de8377fe18 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:09.234876 17682 raft_consensus.cc:493] T e865b8f545e54b7eb0015f6bc1374540 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:09.235127 17682 raft_consensus.cc:515] T e865b8f545e54b7eb0015f6bc1374540 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.235541 16819 catalog_manager.cc:5838] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97df55a0361841319adfae27c9787b2f (127.0.71.65). New cstate: current_term: 1 leader_uuid: "97df55a0361841319adfae27c9787b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:09.236068 17706 ts_tablet_manager.cc:1975] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:09.236328 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e865b8f545e54b7eb0015f6bc1374540" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:09.237627 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e865b8f545e54b7eb0015f6bc1374540" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:09.237989 17398 raft_consensus.cc:493] T 5e9b68ceb78f45edaaebaf7314364446 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:09.238440 17398 raft_consensus.cc:515] T 5e9b68ceb78f45edaaebaf7314364446 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } } peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } }
I20260929 02:01:09.239198 17398 leader_election.cc:290] T 5e9b68ceb78f45edaaebaf7314364446 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233), 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625)
I20260929 02:01:09.239964 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e9b68ceb78f45edaaebaf7314364446" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:09.240234 16984 raft_consensus.cc:2393] T 5e9b68ceb78f45edaaebaf7314364446 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8668afa5b5ca4ebdabc812d0789de7bc in current term 1: Already voted for candidate d103dd4823044faebe46100a0dfbee88 in this term.
I20260929 02:01:09.241935 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.242779 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.242790 17711 raft_consensus.cc:359] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.242965 17711 raft_consensus.cc:385] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.243064 17711 raft_consensus.cc:740] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.243320 17711 consensus_queue.cc:260] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.243275 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.142s user 0.005s sys 0.000s Metrics: {"bytes_written":466,"delete_count":0,"lbm_write_time_us":182,"lbm_writes_lt_1ms":4,"mutex_wait_us":63722,"reinsert_count":0,"update_count":2}
I20260929 02:01:09.243796 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.243896 17711 ts_tablet_manager.cc:1480] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.010s user 0.000s sys 0.000s
I20260929 02:01:09.244542 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.159s user 0.006s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":234,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:09.245184 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.248145 17708 ts_tablet_manager.cc:1975] T 8f328eef30d541c7a61e7d7403e597c7 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:09.248302 17708 log.cc:1199] T 8f328eef30d541c7a61e7d7403e597c7 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/8f328eef30d541c7a61e7d7403e597c7
I20260929 02:01:09.248896 17708 ts_tablet_manager.cc:1996] T 8f328eef30d541c7a61e7d7403e597c7 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
W20260929 02:01:09.237363 17189 leader_election.cc:343] T e865b8f545e54b7eb0015f6bc1374540 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:09.241894 17682 leader_election.cc:290] T e865b8f545e54b7eb0015f6bc1374540 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:09.244331 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5e9b68ceb78f45edaaebaf7314364446" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
W20260929 02:01:09.249948 17186 leader_election.cc:343] T e865b8f545e54b7eb0015f6bc1374540 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:09.250154 17706 log.cc:1199] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/05f61c0da0dd4a20a60cc2b3390cb07c
I20260929 02:01:09.250176 17186 leader_election.cc:304] T e865b8f545e54b7eb0015f6bc1374540 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f
I20260929 02:01:09.250646 17682 raft_consensus.cc:2749] T e865b8f545e54b7eb0015f6bc1374540 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:09.251067 17055 leader_election.cc:304] T 5e9b68ceb78f45edaaebaf7314364446 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc; no voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88
I20260929 02:01:09.251160 17706 ts_tablet_manager.cc:1996] T 05f61c0da0dd4a20a60cc2b3390cb07c P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:09.251554 17398 raft_consensus.cc:3060] T 5e9b68ceb78f45edaaebaf7314364446 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.252583 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 05f61c0da0dd4a20a60cc2b3390cb07c (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:09.255299 17398 raft_consensus.cc:2749] T 5e9b68ceb78f45edaaebaf7314364446 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:09.263501 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 8f328eef30d541c7a61e7d7403e597c7 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:09.265986 17711 tablet_bootstrap.cc:492] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
W20260929 02:01:09.268399 16803 catalog_manager.cc:5030] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): delete failed for tablet a21bdeb06bb24a45945e00b68b17ca5b because the tablet was not found. No further retry: Not found: Tablet not found: a21bdeb06bb24a45945e00b68b17ca5b
I20260929 02:01:09.268476 17708 tablet_replica.cc:333] T 2e0146fb337d43b2921fb5fd34fb95ed P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:09.268800 17708 raft_consensus.cc:2243] T 2e0146fb337d43b2921fb5fd34fb95ed P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:09.269042 17708 raft_consensus.cc:2272] T 2e0146fb337d43b2921fb5fd34fb95ed P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:09.269927 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.045s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":44809}
I20260929 02:01:09.269918 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.047s user 0.005s sys 0.001s Metrics: {"bytes_written":466,"delete_count":0,"lbm_write_time_us":136,"lbm_writes_lt_1ms":4,"mutex_wait_us":34004,"reinsert_count":0,"update_count":2}
I20260929 02:01:09.270186 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.024s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":24409}
I20260929 02:01:09.270478 17711 tablet_bootstrap.cc:654] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:09.270957 17708 ts_tablet_manager.cc:1962] T 2e0146fb337d43b2921fb5fd34fb95ed P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:09.277522 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.086s user 0.008s sys 0.004s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5505,"cfile_init":6,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":89,"dirs.run_cpu_time_us":277,"dirs.run_wall_time_us":4924,"lbm_read_time_us":596,"lbm_reads_lt_1ms":31,"lbm_write_time_us":355,"lbm_writes_lt_1ms":9,"peak_mem_usage":32768,"reinsert_count":0,"update_count":3}
I20260929 02:01:09.279326 17595 tablet_bootstrap.cc:492] T 964849b43d28439aa45e8a51084bcae9 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:09.279711 17595 ts_tablet_manager.cc:1449] T 964849b43d28439aa45e8a51084bcae9 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.113s user 0.004s sys 0.000s
I20260929 02:01:09.280248 17711 tablet_bootstrap.cc:492] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:09.280499 17711 ts_tablet_manager.cc:1449] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.015s user 0.002s sys 0.003s
I20260929 02:01:09.281349 17595 raft_consensus.cc:359] T 964849b43d28439aa45e8a51084bcae9 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.281627 17595 raft_consensus.cc:385] T 964849b43d28439aa45e8a51084bcae9 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.281831 17595 raft_consensus.cc:740] T 964849b43d28439aa45e8a51084bcae9 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.281941 17708 ts_tablet_manager.cc:1975] T 2e0146fb337d43b2921fb5fd34fb95ed P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:09.281977 17682 consensus_queue.cc:1048] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:09.282088 17708 log.cc:1199] T 2e0146fb337d43b2921fb5fd34fb95ed P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/2e0146fb337d43b2921fb5fd34fb95ed
I20260929 02:01:09.282245 17595 consensus_queue.cc:260] T 964849b43d28439aa45e8a51084bcae9 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.282485 17708 ts_tablet_manager.cc:1996] T 2e0146fb337d43b2921fb5fd34fb95ed P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:09.282840 17595 ts_tablet_manager.cc:1480] T 964849b43d28439aa45e8a51084bcae9 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:01:09.283195 17595 tablet_bootstrap.cc:492] T f8dd03b8a32f4ec5917415ee176d272b P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:09.283566 17708 tablet_replica.cc:333] T b402b4aee2a44330a87657f5a9469dc0 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:09.283766 17708 raft_consensus.cc:2243] T b402b4aee2a44330a87657f5a9469dc0 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:09.283700 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.040s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":39538}
I20260929 02:01:09.283951 17708 raft_consensus.cc:2272] T b402b4aee2a44330a87657f5a9469dc0 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:09.283990 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.038s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":38360}
I20260929 02:01:09.284471 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.041s user 0.004s sys 0.001s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":186,"lbm_writes_lt_1ms":4,"mutex_wait_us":34605,"reinsert_count":0,"update_count":1}
I20260929 02:01:09.284576 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 2e0146fb337d43b2921fb5fd34fb95ed (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:09.284914 17708 ts_tablet_manager.cc:1962] T b402b4aee2a44330a87657f5a9469dc0 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:09.285415 17595 tablet_bootstrap.cc:654] T f8dd03b8a32f4ec5917415ee176d272b P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:09.285046 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.064s user 0.005s sys 0.007s Metrics: {"cfile_cache_miss":9,"cfile_cache_miss_bytes":5840,"cfile_init":6,"delete_count":0,"delta_blocks_compacted":3,"delta_iterators_relevant":3,"dirs.queue_time_us":282,"dirs.run_cpu_time_us":276,"dirs.run_wall_time_us":2095,"lbm_read_time_us":562,"lbm_reads_lt_1ms":33,"lbm_write_time_us":338,"lbm_writes_lt_1ms":9,"mutex_wait_us":147,"peak_mem_usage":32768,"reinsert_count":0,"update_count":4}
I20260929 02:01:09.286126 17594 tablet_bootstrap.cc:492] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:09.286293 17594 ts_tablet_manager.cc:1449] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.081s user 0.024s sys 0.014s
I20260929 02:01:09.287086 17594 raft_consensus.cc:359] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.287305 17594 raft_consensus.cc:385] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.287392 17594 raft_consensus.cc:740] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.288890 16819 catalog_manager.cc:5838] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 reported cstate change: term changed from 0 to 1, leader changed from <none> to d103dd4823044faebe46100a0dfbee88 (127.0.71.67). New cstate: current_term: 1 leader_uuid: "d103dd4823044faebe46100a0dfbee88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: HEALTHY } } }
I20260929 02:01:09.290601 17594 consensus_queue.cc:260] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.293023 17594 ts_tablet_manager.cc:1480] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.007s user 0.001s sys 0.002s
I20260929 02:01:09.293337 17594 tablet_bootstrap.cc:492] T 26ea597431574ea9a145a78687442f50 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:09.295595 17594 tablet_bootstrap.cc:654] T 26ea597431574ea9a145a78687442f50 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:09.296175 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.296540 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.296802 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.288496 17711 raft_consensus.cc:359] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.297082 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.297147 17711 raft_consensus.cc:385] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.297282 17711 raft_consensus.cc:740] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.297655 17711 consensus_queue.cc:260] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.298377 17711 ts_tablet_manager.cc:1480] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.018s user 0.002s sys 0.000s
I20260929 02:01:09.298738 17711 tablet_bootstrap.cc:492] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:09.298913 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.299073 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.299149 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.299366 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.299401 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.299621 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.299888 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.300693 17711 tablet_bootstrap.cc:654] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:09.301776 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.302045 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.302134 17398 raft_consensus.cc:493] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:09.302724 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.302294 17398 raft_consensus.cc:515] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.303144 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.210879
I20260929 02:01:09.303393 17398 leader_election.cc:290] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:09.303491 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.210879
I20260929 02:01:09.303570 17708 ts_tablet_manager.cc:1975] T b402b4aee2a44330a87657f5a9469dc0 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:09.303673 17708 log.cc:1199] T b402b4aee2a44330a87657f5a9469dc0 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/b402b4aee2a44330a87657f5a9469dc0
I20260929 02:01:09.303794 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.210879
I20260929 02:01:09.304144 17708 ts_tablet_manager.cc:1996] T b402b4aee2a44330a87657f5a9469dc0 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:09.304227 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.204433
I20260929 02:01:09.304972 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "964849b43d28439aa45e8a51084bcae9" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
W20260929 02:01:09.305725 17056 leader_election.cc:343] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:09.306021 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.305986 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.003s sys 0.002s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":185,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:09.306257 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.306468 17708 tablet_replica.cc:333] T 05f61c0da0dd4a20a60cc2b3390cb07c P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:09.306653 17708 raft_consensus.cc:2243] T 05f61c0da0dd4a20a60cc2b3390cb07c P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:09.306756 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet b402b4aee2a44330a87657f5a9469dc0 (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:09.306821 17708 raft_consensus.cc:2272] T 05f61c0da0dd4a20a60cc2b3390cb07c P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:09.308012 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "964849b43d28439aa45e8a51084bcae9" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:09.308223 17253 raft_consensus.cc:2468] T 964849b43d28439aa45e8a51084bcae9 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:09.308943 17708 ts_tablet_manager.cc:1962] T 05f61c0da0dd4a20a60cc2b3390cb07c P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:09.316756 17055 leader_election.cc:304] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters: 97df55a0361841319adfae27c9787b2f
I20260929 02:01:09.318048 17398 raft_consensus.cc:2804] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:09.318181 17398 raft_consensus.cc:493] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:09.318307 17398 raft_consensus.cc:3060] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.322943 17708 ts_tablet_manager.cc:1975] T 05f61c0da0dd4a20a60cc2b3390cb07c P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:09.323130 17708 log.cc:1199] T 05f61c0da0dd4a20a60cc2b3390cb07c P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/05f61c0da0dd4a20a60cc2b3390cb07c
I20260929 02:01:09.323623 17708 ts_tablet_manager.cc:1996] T 05f61c0da0dd4a20a60cc2b3390cb07c P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:09.324927 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 05f61c0da0dd4a20a60cc2b3390cb07c (table table-2 [id=73fc6290acff43a9b667eafda6683791]) successfully deleted
I20260929 02:01:09.331427 17398 raft_consensus.cc:515] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.332355 17680 raft_consensus.cc:493] T e050cc7cbfa445e3be6081ca25c9ed63 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:09.332647 17680 raft_consensus.cc:515] T e050cc7cbfa445e3be6081ca25c9ed63 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.333506 17680 leader_election.cc:290] T e050cc7cbfa445e3be6081ca25c9ed63 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:09.333756 17398 leader_election.cc:290] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:09.334260 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.037s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":161,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:09.334869 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e050cc7cbfa445e3be6081ca25c9ed63" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:09.334923 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "964849b43d28439aa45e8a51084bcae9" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
W20260929 02:01:09.335525 17189 leader_election.cc:343] T e050cc7cbfa445e3be6081ca25c9ed63 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:09.335891 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e050cc7cbfa445e3be6081ca25c9ed63" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
W20260929 02:01:09.335973 17056 leader_election.cc:343] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:09.336493 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "964849b43d28439aa45e8a51084bcae9" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:09.336740 17253 raft_consensus.cc:3060] T 964849b43d28439aa45e8a51084bcae9 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
W20260929 02:01:09.342484 17186 leader_election.cc:343] T e050cc7cbfa445e3be6081ca25c9ed63 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:09.342660 17186 leader_election.cc:304] T e050cc7cbfa445e3be6081ca25c9ed63 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f
I20260929 02:01:09.343354 17682 raft_consensus.cc:2749] T e050cc7cbfa445e3be6081ca25c9ed63 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:09.343619 17682 consensus_queue.cc:1048] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:09.346593 17543 consensus_queue.cc:1048] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:09.358693 17653 consensus_queue.cc:1048] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:09.360924 17653 raft_consensus.cc:493] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:09.361122 17653 raft_consensus.cc:515] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.361846 17653 leader_election.cc:290] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:09.362623 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16931d61d19f4b1eaed27200e43c5336" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:09.362892 16983 raft_consensus.cc:2468] T 16931d61d19f4b1eaed27200e43c5336 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:09.363626 17056 leader_election.cc:304] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:09.364212 17653 raft_consensus.cc:2804] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:09.364396 17653 raft_consensus.cc:493] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:09.364502 17653 raft_consensus.cc:3060] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.368151 17653 raft_consensus.cc:515] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.369280 17253 raft_consensus.cc:2468] T 964849b43d28439aa45e8a51084bcae9 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:09.373945 16983 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16931d61d19f4b1eaed27200e43c5336" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:09.373965 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16931d61d19f4b1eaed27200e43c5336" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:09.374195 16983 raft_consensus.cc:3060] T 16931d61d19f4b1eaed27200e43c5336 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.377669 17055 leader_election.cc:304] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters: 97df55a0361841319adfae27c9787b2f
W20260929 02:01:09.378238 17055 leader_election.cc:343] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:09.378721 17543 raft_consensus.cc:2804] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:09.378906 17543 raft_consensus.cc:697] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Becoming Leader. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Running, Role: LEADER
I20260929 02:01:09.379246 17543 consensus_queue.cc:237] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.382681 16983 raft_consensus.cc:2468] T 16931d61d19f4b1eaed27200e43c5336 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:09.383806 17056 leader_election.cc:304] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:09.384243 17398 raft_consensus.cc:2804] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:09.384900 17653 leader_election.cc:290] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:09.385309 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16931d61d19f4b1eaed27200e43c5336" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
W20260929 02:01:09.386022 17055 leader_election.cc:343] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:09.387908 17398 raft_consensus.cc:697] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Becoming Leader. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Running, Role: LEADER
I20260929 02:01:09.387953 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.204433
I20260929 02:01:09.388566 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.388549 17398 consensus_queue.cc:237] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.391239 17595 tablet_bootstrap.cc:492] T f8dd03b8a32f4ec5917415ee176d272b P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:09.391429 17595 ts_tablet_manager.cc:1449] T f8dd03b8a32f4ec5917415ee176d272b P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.108s user 0.004s sys 0.000s
I20260929 02:01:09.392535 17595 raft_consensus.cc:359] T f8dd03b8a32f4ec5917415ee176d272b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.392763 17595 raft_consensus.cc:385] T f8dd03b8a32f4ec5917415ee176d272b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.392865 17595 raft_consensus.cc:740] T f8dd03b8a32f4ec5917415ee176d272b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.393136 17595 consensus_queue.cc:260] T f8dd03b8a32f4ec5917415ee176d272b P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.394121 17625 raft_consensus.cc:493] T 800999ad9c144b02b58cf1bf02b6d9e2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:09.394337 17625 raft_consensus.cc:515] T 800999ad9c144b02b58cf1bf02b6d9e2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.395187 17625 leader_election.cc:290] T 800999ad9c144b02b58cf1bf02b6d9e2 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:09.393842 17595 ts_tablet_manager.cc:1480] T f8dd03b8a32f4ec5917415ee176d272b P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:01:09.396304 17595 tablet_bootstrap.cc:492] T 16931d61d19f4b1eaed27200e43c5336 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:09.400017 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "800999ad9c144b02b58cf1bf02b6d9e2" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
W20260929 02:01:09.400789 16919 leader_election.cc:343] T 800999ad9c144b02b58cf1bf02b6d9e2 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:09.401463 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "800999ad9c144b02b58cf1bf02b6d9e2" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
W20260929 02:01:09.402281 16921 leader_election.cc:343] T 800999ad9c144b02b58cf1bf02b6d9e2 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:09.402423 16921 leader_election.cc:304] T 800999ad9c144b02b58cf1bf02b6d9e2 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 97df55a0361841319adfae27c9787b2f; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88
I20260929 02:01:09.403102 17625 raft_consensus.cc:2749] T 800999ad9c144b02b58cf1bf02b6d9e2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:09.404310 17595 tablet_bootstrap.cc:654] T 16931d61d19f4b1eaed27200e43c5336 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:09.407229 16819 catalog_manager.cc:5838] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66). New cstate: current_term: 1 leader_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:09.411437 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.107s user 0.010s sys 0.004s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5616,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":10935,"dirs.run_cpu_time_us":172,"dirs.run_wall_time_us":761,"lbm_read_time_us":727,"lbm_reads_lt_1ms":27,"lbm_write_time_us":346,"lbm_writes_lt_1ms":9,"mutex_wait_us":10824,"peak_mem_usage":32768,"reinsert_count":0,"update_count":3}
I20260929 02:01:09.417050 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.417513 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.418385 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.420398 16819 catalog_manager.cc:5838] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66). New cstate: current_term: 1 leader_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:09.424171 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.424685 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.425076 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.425437 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.427917 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.428267 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.428701 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.434880 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.009s user 0.006s sys 0.001s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":188,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:09.435680 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.436039 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.436375 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.440114 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.445546 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.446520 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.010s user 0.006s sys 0.001s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":190,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:09.447113 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.146826
I20260929 02:01:09.447517 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.146826
I20260929 02:01:09.447799 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.146826
I20260929 02:01:09.450420 17595 tablet_bootstrap.cc:492] T 16931d61d19f4b1eaed27200e43c5336 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:09.450657 17595 ts_tablet_manager.cc:1449] T 16931d61d19f4b1eaed27200e43c5336 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.054s user 0.007s sys 0.000s
I20260929 02:01:09.451654 17595 raft_consensus.cc:359] T 16931d61d19f4b1eaed27200e43c5336 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.451952 17595 raft_consensus.cc:385] T 16931d61d19f4b1eaed27200e43c5336 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.452051 17595 raft_consensus.cc:740] T 16931d61d19f4b1eaed27200e43c5336 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.454574 17769 raft_consensus.cc:493] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:09.454777 17769 raft_consensus.cc:515] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.455595 17769 leader_election.cc:290] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:09.456670 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.456986 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.456988 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "273fd95488284ef88c2272052f0de976" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:09.457219 16986 raft_consensus.cc:2468] T 273fd95488284ef88c2272052f0de976 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:09.457221 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "273fd95488284ef88c2272052f0de976" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:09.457633 17056 leader_election.cc:304] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:09.457970 17398 raft_consensus.cc:2804] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:09.458213 17398 raft_consensus.cc:493] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:09.458417 17398 raft_consensus.cc:3060] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
W20260929 02:01:09.459009 17055 leader_election.cc:343] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:09.459476 17543 consensus_queue.cc:1048] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:09.464238 17625 raft_consensus.cc:493] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:09.464535 17625 raft_consensus.cc:515] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.465476 17625 leader_election.cc:290] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:09.465673 17398 raft_consensus.cc:515] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.466825 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26ea597431574ea9a145a78687442f50" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:09.466868 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "273fd95488284ef88c2272052f0de976" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:09.467056 16986 raft_consensus.cc:3060] T 273fd95488284ef88c2272052f0de976 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
W20260929 02:01:09.467537 16919 leader_election.cc:343] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:09.468004 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26ea597431574ea9a145a78687442f50" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:09.468217 17253 raft_consensus.cc:2468] T 26ea597431574ea9a145a78687442f50 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:09.468628 16921 leader_election.cc:304] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc
I20260929 02:01:09.469285 17625 raft_consensus.cc:2804] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:09.469409 17625 raft_consensus.cc:493] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:09.469538 17625 raft_consensus.cc:3060] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.470395 16986 raft_consensus.cc:2468] T 273fd95488284ef88c2272052f0de976 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:09.470963 17056 leader_election.cc:304] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:09.471243 17543 raft_consensus.cc:2804] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:09.471436 17543 raft_consensus.cc:697] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Becoming Leader. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Running, Role: LEADER
I20260929 02:01:09.471783 17543 consensus_queue.cc:237] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.476668 16819 catalog_manager.cc:5838] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66). New cstate: current_term: 1 leader_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:09.476935 17625 raft_consensus.cc:515] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.478950 17398 leader_election.cc:290] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:09.479939 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.032s user 0.010s sys 0.003s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5697,"cfile_init":6,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":100,"lbm_read_time_us":672,"lbm_reads_lt_1ms":31,"lbm_write_time_us":413,"lbm_writes_lt_1ms":9,"peak_mem_usage":32768,"reinsert_count":0,"update_count":2}
I20260929 02:01:09.489593 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "273fd95488284ef88c2272052f0de976" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
W20260929 02:01:09.490407 17055 leader_election.cc:343] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:09.491400 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.491353 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26ea597431574ea9a145a78687442f50" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:09.491760 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.492095 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
W20260929 02:01:09.492298 16919 leader_election.cc:343] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:09.492403 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.493419 17625 leader_election.cc:290] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Requested vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:09.493816 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "26ea597431574ea9a145a78687442f50" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:09.494029 17253 raft_consensus.cc:3060] T 26ea597431574ea9a145a78687442f50 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.494724 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.495020 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.495271 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.498431 17253 raft_consensus.cc:2468] T 26ea597431574ea9a145a78687442f50 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:09.499055 16921 leader_election.cc:304] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc
I20260929 02:01:09.499704 17625 raft_consensus.cc:2804] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:09.499920 17625 raft_consensus.cc:697] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Becoming Leader. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Running, Role: LEADER
I20260929 02:01:09.500173 17625 consensus_queue.cc:237] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.500540 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.008s user 0.006s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":189,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:09.513854 17624 consensus_queue.cc:1048] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:09.544092 17398 consensus_queue.cc:1048] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:09.573504 17624 consensus_queue.cc:1048] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:09.576961 17624 consensus_queue.cc:1048] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
W20260929 02:01:09.594116 16921 consensus_peers.cc:597] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f -> Peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Couldn't send request to peer d103dd4823044faebe46100a0dfbee88. Error code: TABLET_NOT_RUNNING (12). Status: Illegal state: Tablet not RUNNING: INITIALIZED. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:01:09.511955 17595 consensus_queue.cc:260] T 16931d61d19f4b1eaed27200e43c5336 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.598393 17674 raft_consensus.cc:493] T 16931d61d19f4b1eaed27200e43c5336 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:09.598640 17674 raft_consensus.cc:515] T 16931d61d19f4b1eaed27200e43c5336 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.600517 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16931d61d19f4b1eaed27200e43c5336" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:09.600651 17674 leader_election.cc:290] T 16931d61d19f4b1eaed27200e43c5336 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:09.601866 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "16931d61d19f4b1eaed27200e43c5336" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:09.603394 17595 ts_tablet_manager.cc:1480] T 16931d61d19f4b1eaed27200e43c5336 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.153s user 0.002s sys 0.001s
I20260929 02:01:09.545024 16816 catalog_manager.cc:5838] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97df55a0361841319adfae27c9787b2f (127.0.71.65). New cstate: current_term: 1 leader_uuid: "97df55a0361841319adfae27c9787b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:09.606240 16986 raft_consensus.cc:2393] T 16931d61d19f4b1eaed27200e43c5336 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate d103dd4823044faebe46100a0dfbee88 in current term 1: Already voted for candidate 8668afa5b5ca4ebdabc812d0789de7bc in this term.
I20260929 02:01:09.606812 17189 leader_election.cc:304] T 16931d61d19f4b1eaed27200e43c5336 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f
I20260929 02:01:09.607317 17680 raft_consensus.cc:3060] T 16931d61d19f4b1eaed27200e43c5336 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.611443 17680 raft_consensus.cc:2749] T 16931d61d19f4b1eaed27200e43c5336 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:09.614288 17780 consensus_queue.cc:1048] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:09.627329 17780 consensus_queue.cc:1048] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:09.639827 17682 raft_consensus.cc:493] T f8dd03b8a32f4ec5917415ee176d272b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:09.640023 17682 raft_consensus.cc:515] T f8dd03b8a32f4ec5917415ee176d272b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.640667 17682 leader_election.cc:290] T f8dd03b8a32f4ec5917415ee176d272b P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:09.643077 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8dd03b8a32f4ec5917415ee176d272b" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
W20260929 02:01:09.643754 17189 leader_election.cc:343] T f8dd03b8a32f4ec5917415ee176d272b P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:09.644721 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8dd03b8a32f4ec5917415ee176d272b" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
W20260929 02:01:09.645376 17186 leader_election.cc:343] T f8dd03b8a32f4ec5917415ee176d272b P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:09.646540 17595 tablet_bootstrap.cc:492] T 273fd95488284ef88c2272052f0de976 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:09.648953 17595 tablet_bootstrap.cc:654] T 273fd95488284ef88c2272052f0de976 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:09.645634 17186 leader_election.cc:304] T f8dd03b8a32f4ec5917415ee176d272b P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f
I20260929 02:01:09.649863 17680 raft_consensus.cc:2749] T f8dd03b8a32f4ec5917415ee176d272b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:09.653219 16986 raft_consensus.cc:3060] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.658418 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.354s user 0.014s sys 0.000s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5623,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":96,"lbm_read_time_us":1823,"lbm_reads_lt_1ms":27,"lbm_write_time_us":393,"lbm_writes_lt_1ms":9,"mutex_wait_us":4361,"peak_mem_usage":32768,"reinsert_count":0,"update_count":2}
I20260929 02:01:09.660024 17594 tablet_bootstrap.cc:492] T 26ea597431574ea9a145a78687442f50 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:09.660203 17594 ts_tablet_manager.cc:1449] T 26ea597431574ea9a145a78687442f50 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.367s user 0.065s sys 0.186s
I20260929 02:01:09.661094 17711 tablet_bootstrap.cc:492] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:09.661298 17711 ts_tablet_manager.cc:1449] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.363s user 0.095s sys 0.157s
I20260929 02:01:09.662213 17711 raft_consensus.cc:359] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.662412 17711 raft_consensus.cc:385] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.662508 17711 raft_consensus.cc:740] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.662734 17711 consensus_queue.cc:260] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.663709 17711 ts_tablet_manager.cc:1480] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.002s user 0.001s sys 0.002s
I20260929 02:01:09.663805 17594 raft_consensus.cc:359] T 26ea597431574ea9a145a78687442f50 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.663995 17711 tablet_bootstrap.cc:492] T 964849b43d28439aa45e8a51084bcae9 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:09.664018 17594 raft_consensus.cc:385] T 26ea597431574ea9a145a78687442f50 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.664094 17594 raft_consensus.cc:740] T 26ea597431574ea9a145a78687442f50 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.664350 17594 consensus_queue.cc:260] T 26ea597431574ea9a145a78687442f50 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.664840 17594 ts_tablet_manager.cc:1480] T 26ea597431574ea9a145a78687442f50 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260929 02:01:09.665126 17594 tablet_bootstrap.cc:492] T e865b8f545e54b7eb0015f6bc1374540 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:09.666065 17595 tablet_bootstrap.cc:492] T 273fd95488284ef88c2272052f0de976 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:09.666249 17711 tablet_bootstrap.cc:654] T 964849b43d28439aa45e8a51084bcae9 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:09.666297 17595 ts_tablet_manager.cc:1449] T 273fd95488284ef88c2272052f0de976 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.020s user 0.004s sys 0.000s
I20260929 02:01:09.666878 17594 tablet_bootstrap.cc:654] T e865b8f545e54b7eb0015f6bc1374540 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:09.667158 17595 raft_consensus.cc:359] T 273fd95488284ef88c2272052f0de976 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.667364 17595 raft_consensus.cc:385] T 273fd95488284ef88c2272052f0de976 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.667461 17595 raft_consensus.cc:740] T 273fd95488284ef88c2272052f0de976 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.667753 17595 consensus_queue.cc:260] T 273fd95488284ef88c2272052f0de976 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.668419 17595 ts_tablet_manager.cc:1480] T 273fd95488284ef88c2272052f0de976 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 02:01:09.668820 17595 tablet_bootstrap.cc:492] T e5aa0d7aa540479c89ae9b8b61d89c4e P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:09.671994 17595 tablet_bootstrap.cc:654] T e5aa0d7aa540479c89ae9b8b61d89c4e P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:09.679558 17779 raft_consensus.cc:493] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:09.679817 17779 raft_consensus.cc:515] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.680900 17779 leader_election.cc:290] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 2 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:09.681808 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8c4b957e16467fa6bd6e9c317b1b1f" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:09.682045 17118 raft_consensus.cc:2468] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:09.682659 16919 leader_election.cc:304] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 2 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:09.683095 17779 raft_consensus.cc:2804] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader pre-election won for term 2
I20260929 02:01:09.683229 17779 raft_consensus.cc:493] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:09.683346 17779 raft_consensus.cc:3060] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:01:09.687602 17779 raft_consensus.cc:515] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f [term 2 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.688980 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8c4b957e16467fa6bd6e9c317b1b1f" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:09.689220 17118 raft_consensus.cc:3060] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:01:09.692655 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.694113 17118 raft_consensus.cc:2468] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 8668afa5b5ca4ebdabc812d0789de7bc [term 2 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 2.
I20260929 02:01:09.694696 16919 leader_election.cc:304] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 2 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:09.695115 17624 raft_consensus.cc:2804] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f [term 2 FOLLOWER]: Leader election won for term 2
I20260929 02:01:09.695277 17624 raft_consensus.cc:697] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f [term 2 LEADER]: Becoming Leader. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Running, Role: LEADER
I20260929 02:01:09.695590 17624 consensus_queue.cc:237] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 2, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.698266 17398 consensus_queue.cc:1048] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:09.700228 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.700320 17779 leader_election.cc:290] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 2 election: Requested vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:09.700659 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.701223 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8c4b957e16467fa6bd6e9c317b1b1f" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:09.702366 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.702858 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.199401
I20260929 02:01:09.703219 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.199401
I20260929 02:01:09.703570 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.199401
I20260929 02:01:09.703969 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.704252 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.705140 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:09.705672 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.706823 17251 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "2e8c4b957e16467fa6bd6e9c317b1b1f" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 2 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:09.706930 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.288s user 0.006s sys 0.000s Metrics: {"bytes_written":480,"delete_count":0,"lbm_write_time_us":145,"lbm_writes_lt_1ms":4,"mutex_wait_us":241867,"reinsert_count":0,"update_count":3}
I20260929 02:01:09.707494 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.209294
I20260929 02:01:09.707778 17680 consensus_queue.cc:1048] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:09.709767 17543 consensus_queue.cc:1048] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:09.710040 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.286s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":286089}
I20260929 02:01:09.713133 17595 tablet_bootstrap.cc:492] T e5aa0d7aa540479c89ae9b8b61d89c4e P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:09.713320 17595 ts_tablet_manager.cc:1449] T e5aa0d7aa540479c89ae9b8b61d89c4e P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.045s user 0.004s sys 0.006s
I20260929 02:01:09.713694 16986 raft_consensus.cc:1240] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f [term 2 LEADER]: Rejecting Update request from peer d103dd4823044faebe46100a0dfbee88 for earlier term 1. Current term is 2. Ops: [1.1-1.1]
I20260929 02:01:09.714864 17682 consensus_queue.cc:1059] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [LEADER]: Peer responded invalid term: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: INVALID_TERM, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:09.715099 17682 consensus_queue.cc:1243] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [LEADER]: Peer 97df55a0361841319adfae27c9787b2f log is divergent from this leader: its last log entry 2.1 is not in this leader's log and it has not received anything from this leader yet. Falling back to committed index 0
I20260929 02:01:09.715230 17682 raft_consensus.cc:3055] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Stepping down as leader of term 1
I20260929 02:01:09.715329 17682 raft_consensus.cc:740] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Running, Role: LEADER
I20260929 02:01:09.715626 17682 consensus_queue.cc:260] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 1.1, Last appended by leader: 1, Current term: 1, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.715962 17682 raft_consensus.cc:3060] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Advancing to term 2
I20260929 02:01:09.716820 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.298s user 0.006s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":194,"lbm_writes_lt_1ms":4,"mutex_wait_us":280949,"reinsert_count":0,"update_count":1}
I20260929 02:01:09.720546 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.020s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":181,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:09.720619 17595 raft_consensus.cc:359] T e5aa0d7aa540479c89ae9b8b61d89c4e P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.720920 17595 raft_consensus.cc:385] T e5aa0d7aa540479c89ae9b8b61d89c4e P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.721046 17595 raft_consensus.cc:740] T e5aa0d7aa540479c89ae9b8b61d89c4e P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.721424 17595 consensus_queue.cc:260] T e5aa0d7aa540479c89ae9b8b61d89c4e P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.721310 16816 catalog_manager.cc:5838] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f reported cstate change: term changed from 1 to 2, leader changed from d103dd4823044faebe46100a0dfbee88 (127.0.71.67) to 97df55a0361841319adfae27c9787b2f (127.0.71.65). New cstate: current_term: 2 leader_uuid: "97df55a0361841319adfae27c9787b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:09.722138 17595 ts_tablet_manager.cc:1480] T e5aa0d7aa540479c89ae9b8b61d89c4e P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.009s user 0.003s sys 0.000s
I20260929 02:01:09.722479 17595 tablet_bootstrap.cc:492] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:09.724323 17595 tablet_bootstrap.cc:654] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:09.732187 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.028s user 0.011s sys 0.000s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5771,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":5925,"lbm_read_time_us":686,"lbm_reads_lt_1ms":27,"lbm_write_time_us":374,"lbm_writes_lt_1ms":9,"mutex_wait_us":5926,"peak_mem_usage":32768,"reinsert_count":0,"update_count":2}
I20260929 02:01:09.733659 17595 tablet_bootstrap.cc:492] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:09.733855 17595 ts_tablet_manager.cc:1449] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.011s user 0.007s sys 0.002s
I20260929 02:01:09.734620 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.734858 17595 raft_consensus.cc:359] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.735029 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.735092 17595 raft_consensus.cc:385] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.735189 17595 raft_consensus.cc:740] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.735484 17595 consensus_queue.cc:260] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.736138 17595 ts_tablet_manager.cc:1480] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 02:01:09.743403 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.743741 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.743993 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.744248 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.745939 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.746183 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.746380 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.751086 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":126,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:09.761726 17594 tablet_bootstrap.cc:492] T e865b8f545e54b7eb0015f6bc1374540 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:09.761907 17594 ts_tablet_manager.cc:1449] T e865b8f545e54b7eb0015f6bc1374540 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.097s user 0.024s sys 0.038s
I20260929 02:01:09.762104 17711 tablet_bootstrap.cc:492] T 964849b43d28439aa45e8a51084bcae9 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:09.762274 17711 ts_tablet_manager.cc:1449] T 964849b43d28439aa45e8a51084bcae9 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.098s user 0.009s sys 0.028s
I20260929 02:01:09.762626 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.023s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":23104}
I20260929 02:01:09.762826 17594 raft_consensus.cc:359] T e865b8f545e54b7eb0015f6bc1374540 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.762928 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.023s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":23289}
I20260929 02:01:09.763048 17594 raft_consensus.cc:385] T e865b8f545e54b7eb0015f6bc1374540 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.762980 17711 raft_consensus.cc:359] T 964849b43d28439aa45e8a51084bcae9 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.763161 17594 raft_consensus.cc:740] T e865b8f545e54b7eb0015f6bc1374540 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.763167 17711 raft_consensus.cc:385] T 964849b43d28439aa45e8a51084bcae9 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.763255 17711 raft_consensus.cc:740] T 964849b43d28439aa45e8a51084bcae9 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.763476 17711 consensus_queue.cc:260] T 964849b43d28439aa45e8a51084bcae9 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.763468 17594 consensus_queue.cc:260] T e865b8f545e54b7eb0015f6bc1374540 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.763954 17711 ts_tablet_manager.cc:1480] T 964849b43d28439aa45e8a51084bcae9 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.002s user 0.001s sys 0.001s
I20260929 02:01:09.764261 17711 tablet_bootstrap.cc:492] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:09.765570 17594 ts_tablet_manager.cc:1480] T e865b8f545e54b7eb0015f6bc1374540 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260929 02:01:09.765864 17594 tablet_bootstrap.cc:492] T f8dd03b8a32f4ec5917415ee176d272b P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:09.767571 17711 tablet_bootstrap.cc:654] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:09.767761 17594 tablet_bootstrap.cc:654] T f8dd03b8a32f4ec5917415ee176d272b P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:09.769105 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.076s user 0.006s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":176,"lbm_writes_lt_1ms":4,"mutex_wait_us":53108,"reinsert_count":0,"update_count":1}
I20260929 02:01:09.777515 17594 tablet_bootstrap.cc:492] T f8dd03b8a32f4ec5917415ee176d272b P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:09.777694 17594 ts_tablet_manager.cc:1449] T f8dd03b8a32f4ec5917415ee176d272b P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.005s
I20260929 02:01:09.778509 17594 raft_consensus.cc:359] T f8dd03b8a32f4ec5917415ee176d272b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.778719 17594 raft_consensus.cc:385] T f8dd03b8a32f4ec5917415ee176d272b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.778805 17594 raft_consensus.cc:740] T f8dd03b8a32f4ec5917415ee176d272b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.779261 17594 consensus_queue.cc:260] T f8dd03b8a32f4ec5917415ee176d272b P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.779223 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.069s user 0.013s sys 0.000s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5741,"cfile_init":6,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":86,"dirs.run_cpu_time_us":266,"dirs.run_wall_time_us":5804,"lbm_read_time_us":724,"lbm_reads_lt_1ms":31,"lbm_write_time_us":299,"lbm_writes_lt_1ms":9,"peak_mem_usage":32768,"reinsert_count":0,"update_count":4}
I20260929 02:01:09.781358 17594 ts_tablet_manager.cc:1480] T f8dd03b8a32f4ec5917415ee176d272b P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.004s user 0.001s sys 0.002s
I20260929 02:01:09.781677 17594 tablet_bootstrap.cc:492] T e050cc7cbfa445e3be6081ca25c9ed63 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:09.783280 17711 tablet_bootstrap.cc:492] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:09.783416 17711 ts_tablet_manager.cc:1449] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.019s user 0.008s sys 0.000s
I20260929 02:01:09.784065 17711 raft_consensus.cc:359] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.784237 17711 raft_consensus.cc:385] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.784310 17711 raft_consensus.cc:740] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.784549 17711 consensus_queue.cc:260] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.785434 17594 tablet_bootstrap.cc:654] T e050cc7cbfa445e3be6081ca25c9ed63 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:09.786595 17711 ts_tablet_manager.cc:1480] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:01:09.786868 17711 tablet_bootstrap.cc:492] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:09.789063 17711 tablet_bootstrap.cc:654] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:09.790848 17779 raft_consensus.cc:493] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:09.791030 17779 raft_consensus.cc:515] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.791829 17779 leader_election.cc:290] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:09.792613 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e050cc7cbfa445e3be6081ca25c9ed63" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:09.792866 17253 raft_consensus.cc:2468] T e050cc7cbfa445e3be6081ca25c9ed63 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:09.793390 16921 leader_election.cc:304] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:09.793407 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e050cc7cbfa445e3be6081ca25c9ed63" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:09.793716 17779 raft_consensus.cc:2804] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:09.793823 17779 raft_consensus.cc:493] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:09.793901 17779 raft_consensus.cc:3060] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
W20260929 02:01:09.794214 16919 leader_election.cc:343] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:09.797544 17543 consensus_queue.cc:1048] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:09.797582 17779 raft_consensus.cc:515] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.798671 17250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e050cc7cbfa445e3be6081ca25c9ed63" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:09.798853 17250 raft_consensus.cc:3060] T e050cc7cbfa445e3be6081ca25c9ed63 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.799492 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e050cc7cbfa445e3be6081ca25c9ed63" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
W20260929 02:01:09.800006 16919 leader_election.cc:343] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:09.802431 17250 raft_consensus.cc:2468] T e050cc7cbfa445e3be6081ca25c9ed63 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:09.804498 16921 leader_election.cc:304] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc
I20260929 02:01:09.804870 17624 raft_consensus.cc:2804] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:09.805086 17624 raft_consensus.cc:697] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Becoming Leader. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Running, Role: LEADER
I20260929 02:01:09.805419 17624 consensus_queue.cc:237] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.805974 17543 consensus_queue.cc:1048] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:09.808671 17779 leader_election.cc:290] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Requested vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:09.822088 16819 catalog_manager.cc:5838] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97df55a0361841319adfae27c9787b2f (127.0.71.65). New cstate: current_term: 1 leader_uuid: "97df55a0361841319adfae27c9787b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:09.828670 17543 consensus_queue.cc:1048] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:09.836220 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.836602 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.837983 17594 tablet_bootstrap.cc:492] T e050cc7cbfa445e3be6081ca25c9ed63 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:09.838155 17594 ts_tablet_manager.cc:1449] T e050cc7cbfa445e3be6081ca25c9ed63 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.057s user 0.000s sys 0.012s
I20260929 02:01:09.838552 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.840653 17711 tablet_bootstrap.cc:492] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:09.840824 17711 ts_tablet_manager.cc:1449] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.054s user 0.023s sys 0.026s
I20260929 02:01:09.841553 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.841729 17711 raft_consensus.cc:359] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.841869 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.196977
I20260929 02:01:09.841951 17711 raft_consensus.cc:385] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.842028 17711 raft_consensus.cc:740] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.842211 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.196977
I20260929 02:01:09.842278 17711 consensus_queue.cc:260] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.842442 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.842935 17711 ts_tablet_manager.cc:1480] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:01:09.841866 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.844168 17594 raft_consensus.cc:359] T e050cc7cbfa445e3be6081ca25c9ed63 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.844365 17594 raft_consensus.cc:385] T e050cc7cbfa445e3be6081ca25c9ed63 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.844468 17594 raft_consensus.cc:740] T e050cc7cbfa445e3be6081ca25c9ed63 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.844703 17594 consensus_queue.cc:260] T e050cc7cbfa445e3be6081ca25c9ed63 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.846863 17594 ts_tablet_manager.cc:1480] T e050cc7cbfa445e3be6081ca25c9ed63 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.009s user 0.000s sys 0.003s
I20260929 02:01:09.847165 17594 tablet_bootstrap.cc:492] T 800999ad9c144b02b58cf1bf02b6d9e2 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:09.858281 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.858618 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.858685 17624 raft_consensus.cc:493] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:09.858891 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.858860 17624 raft_consensus.cc:515] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.859120 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.859582 17624 leader_election.cc:290] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:09.860819 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.861088 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.861351 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.861902 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.196977
I20260929 02:01:09.862154 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.196977
I20260929 02:01:09.862461 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.196977
I20260929 02:01:09.862833 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.863168 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.863435 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:09.865052 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.865048 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e865b8f545e54b7eb0015f6bc1374540" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:09.865298 17118 raft_consensus.cc:2468] T e865b8f545e54b7eb0015f6bc1374540 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:09.865840 16919 leader_election.cc:304] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:09.866112 17624 raft_consensus.cc:2804] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:09.866223 17624 raft_consensus.cc:493] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:09.866338 17624 raft_consensus.cc:3060] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.866348 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.029s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":153,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:09.866941 17250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e865b8f545e54b7eb0015f6bc1374540" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:09.867153 17250 raft_consensus.cc:2468] T e865b8f545e54b7eb0015f6bc1374540 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:09.869349 17624 raft_consensus.cc:515] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.870040 17624 leader_election.cc:290] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Requested vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:09.870697 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e865b8f545e54b7eb0015f6bc1374540" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:09.870927 17118 raft_consensus.cc:3060] T e865b8f545e54b7eb0015f6bc1374540 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.873520 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.014s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":138,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:09.874435 17118 raft_consensus.cc:2468] T e865b8f545e54b7eb0015f6bc1374540 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:09.874958 16919 leader_election.cc:304] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:09.875324 17624 raft_consensus.cc:2804] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:09.875622 17624 raft_consensus.cc:697] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Becoming Leader. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Running, Role: LEADER
I20260929 02:01:09.875494 17250 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e865b8f545e54b7eb0015f6bc1374540" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:09.875924 17624 consensus_queue.cc:237] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.876137 17250 raft_consensus.cc:3060] T e865b8f545e54b7eb0015f6bc1374540 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.877390 17594 tablet_bootstrap.cc:654] T 800999ad9c144b02b58cf1bf02b6d9e2 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:09.880430 16986 raft_consensus.cc:3060] T 964849b43d28439aa45e8a51084bcae9 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.880945 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.018s user 0.010s sys 0.000s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5845,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":1341,"dirs.run_cpu_time_us":151,"dirs.run_wall_time_us":678,"lbm_read_time_us":489,"lbm_reads_lt_1ms":27,"lbm_write_time_us":295,"lbm_writes_lt_1ms":9,"mutex_wait_us":5973,"peak_mem_usage":32768,"reinsert_count":0,"update_count":2}
I20260929 02:01:09.883138 17594 tablet_bootstrap.cc:492] T 800999ad9c144b02b58cf1bf02b6d9e2 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:09.883356 17594 ts_tablet_manager.cc:1449] T 800999ad9c144b02b58cf1bf02b6d9e2 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.036s user 0.001s sys 0.003s
I20260929 02:01:09.883988 17398 consensus_queue.cc:1048] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:09.884263 17594 raft_consensus.cc:359] T 800999ad9c144b02b58cf1bf02b6d9e2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.884519 17594 raft_consensus.cc:385] T 800999ad9c144b02b58cf1bf02b6d9e2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:09.884631 17594 raft_consensus.cc:740] T 800999ad9c144b02b58cf1bf02b6d9e2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:09.884898 17594 consensus_queue.cc:260] T 800999ad9c144b02b58cf1bf02b6d9e2 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:09.885887 17594 ts_tablet_manager.cc:1480] T 800999ad9c144b02b58cf1bf02b6d9e2 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.002s user 0.002s sys 0.000s
I20260929 02:01:09.887480 16819 catalog_manager.cc:5838] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97df55a0361841319adfae27c9787b2f (127.0.71.65). New cstate: current_term: 1 leader_uuid: "97df55a0361841319adfae27c9787b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:09.889554 17118 raft_consensus.cc:3060] T 26ea597431574ea9a145a78687442f50 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.892637 17250 raft_consensus.cc:2468] T e865b8f545e54b7eb0015f6bc1374540 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:09.896534 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.054s user 0.011s sys 0.000s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5845,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":5630,"dirs.run_cpu_time_us":139,"dirs.run_wall_time_us":1645,"lbm_read_time_us":534,"lbm_reads_lt_1ms":27,"lbm_write_time_us":336,"lbm_writes_lt_1ms":9,"mutex_wait_us":20745,"peak_mem_usage":32768,"reinsert_count":0,"update_count":2}
I20260929 02:01:09.898330 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.898633 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.898939 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.899181 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.899449 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.900130 17624 consensus_queue.cc:1048] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:09.902652 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.902953 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.903206 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.903442 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:09.903687 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.903924 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.904098 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:09.909888 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.006s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":185,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:09.912276 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.013s user 0.004s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":152,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:09.916244 17253 raft_consensus.cc:3060] T 273fd95488284ef88c2272052f0de976 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:09.916646 17779 consensus_queue.cc:1048] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:09.920789 17769 consensus_queue.cc:1048] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:09.928786 17653 consensus_queue.cc:1048] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:10.016599 17653 raft_consensus.cc:493] T e050cc7cbfa445e3be6081ca25c9ed63 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:10.016892 17653 raft_consensus.cc:515] T e050cc7cbfa445e3be6081ca25c9ed63 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.017630 17653 leader_election.cc:290] T e050cc7cbfa445e3be6081ca25c9ed63 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:10.018224 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e050cc7cbfa445e3be6081ca25c9ed63" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:10.018321 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e050cc7cbfa445e3be6081ca25c9ed63" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:10.018550 17253 raft_consensus.cc:2393] T e050cc7cbfa445e3be6081ca25c9ed63 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader pre-election vote request: Denying vote to candidate 8668afa5b5ca4ebdabc812d0789de7bc in current term 1: Already voted for candidate 97df55a0361841319adfae27c9787b2f in this term.
I20260929 02:01:10.019017 17055 leader_election.cc:304] T e050cc7cbfa445e3be6081ca25c9ed63 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc; no voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88
I20260929 02:01:10.019309 17653 raft_consensus.cc:3060] T e050cc7cbfa445e3be6081ca25c9ed63 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:10.021296 17653 raft_consensus.cc:2749] T e050cc7cbfa445e3be6081ca25c9ed63 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:10.037349 17779 raft_consensus.cc:493] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:10.037613 17779 raft_consensus.cc:515] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.038275 17779 leader_election.cc:290] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:10.038774 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8dd03b8a32f4ec5917415ee176d272b" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:10.038939 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8dd03b8a32f4ec5917415ee176d272b" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:10.039088 17118 raft_consensus.cc:2468] T f8dd03b8a32f4ec5917415ee176d272b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:10.039177 17253 raft_consensus.cc:2468] T f8dd03b8a32f4ec5917415ee176d272b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:10.039568 16919 leader_election.cc:304] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:10.039871 17779 raft_consensus.cc:2804] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:10.039958 17779 raft_consensus.cc:493] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:10.040071 17779 raft_consensus.cc:3060] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:10.042223 17779 raft_consensus.cc:515] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.042841 17779 leader_election.cc:290] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Requested vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:10.043457 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8dd03b8a32f4ec5917415ee176d272b" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:10.043455 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "f8dd03b8a32f4ec5917415ee176d272b" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:10.043649 17253 raft_consensus.cc:3060] T f8dd03b8a32f4ec5917415ee176d272b P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:10.043668 17118 raft_consensus.cc:3060] T f8dd03b8a32f4ec5917415ee176d272b P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:10.046658 17253 raft_consensus.cc:2468] T f8dd03b8a32f4ec5917415ee176d272b P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:10.046698 17118 raft_consensus.cc:2468] T f8dd03b8a32f4ec5917415ee176d272b P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:10.047165 16919 leader_election.cc:304] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:10.047489 17779 raft_consensus.cc:2804] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:10.047650 17779 raft_consensus.cc:697] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Becoming Leader. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Running, Role: LEADER
I20260929 02:01:10.047937 17779 consensus_queue.cc:237] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.051185 16816 catalog_manager.cc:5838] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97df55a0361841319adfae27c9787b2f (127.0.71.65). New cstate: current_term: 1 leader_uuid: "97df55a0361841319adfae27c9787b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:10.057518 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.057834 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.058116 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.058472 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.195787
I20260929 02:01:10.058734 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.195787
I20260929 02:01:10.058991 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:10.058991 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.059338 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.059648 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.060003 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.062009 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.004s user 0.003s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":142,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:10.062618 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.063046 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.195787
I20260929 02:01:10.063308 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.195787
I20260929 02:01:10.063653 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.064023 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:10.064368 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.068629 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.009s user 0.004s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":136,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:10.070828 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.012s user 0.005s sys 0.002s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5882,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":188,"lbm_read_time_us":323,"lbm_reads_lt_1ms":27,"lbm_write_time_us":224,"lbm_writes_lt_1ms":9,"mutex_wait_us":3218,"peak_mem_usage":32768,"reinsert_count":0,"update_count":2}
I20260929 02:01:10.072433 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.009s user 0.007s sys 0.000s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":5882,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":163,"lbm_read_time_us":296,"lbm_reads_lt_1ms":27,"lbm_write_time_us":212,"lbm_writes_lt_1ms":9,"mutex_wait_us":659,"peak_mem_usage":32768,"reinsert_count":0,"update_count":2}
I20260929 02:01:10.077319 17682 raft_consensus.cc:493] T e5aa0d7aa540479c89ae9b8b61d89c4e P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:10.077533 17682 raft_consensus.cc:515] T e5aa0d7aa540479c89ae9b8b61d89c4e P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.078290 17682 leader_election.cc:290] T e5aa0d7aa540479c89ae9b8b61d89c4e P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:10.078950 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5aa0d7aa540479c89ae9b8b61d89c4e" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:10.078999 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "e5aa0d7aa540479c89ae9b8b61d89c4e" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:10.079618 17186 leader_election.cc:304] T e5aa0d7aa540479c89ae9b8b61d89c4e P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f
I20260929 02:01:10.079989 17682 raft_consensus.cc:2749] T e5aa0d7aa540479c89ae9b8b61d89c4e P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:10.084668 17118 raft_consensus.cc:1275] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 8668afa5b5ca4ebdabc812d0789de7bc [term 2 FOLLOWER]: Refusing update from remote peer 97df55a0361841319adfae27c9787b2f: Log matching property violated. Preceding OpId in replica: term: 0 index: 0. Preceding OpId from leader: term: 2 index: 1. (index mismatch)
I20260929 02:01:10.085220 17779 consensus_queue.cc:1048] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:10.089373 17253 raft_consensus.cc:1275] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [term 2 FOLLOWER]: Refusing update from remote peer 97df55a0361841319adfae27c9787b2f: Log matching property violated. Preceding OpId in replica: term: 1 index: 1. Preceding OpId from leader: term: 2 index: 1. (term mismatch)
I20260929 02:01:10.089514 17253 pending_rounds.cc:85] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88: Aborting all ops after (but not including) 0
I20260929 02:01:10.089589 17253 pending_rounds.cc:107] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88: Aborting uncommitted NO_OP operation due to leader change: 1.1
I20260929 02:01:10.089726 17253 raft_consensus.cc:2889] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [term 2 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted by new leader
I20260929 02:01:10.090474 17779 consensus_queue.cc:1048] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:10.101822 17253 raft_consensus.cc:3060] T e5aa0d7aa540479c89ae9b8b61d89c4e P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:10.121753 17682 raft_consensus.cc:493] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:10.121963 17682 raft_consensus.cc:515] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.122571 17682 leader_election.cc:290] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:10.123090 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "800999ad9c144b02b58cf1bf02b6d9e2" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:10.123383 16984 raft_consensus.cc:2468] T 800999ad9c144b02b58cf1bf02b6d9e2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 0.
I20260929 02:01:10.123647 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "800999ad9c144b02b58cf1bf02b6d9e2" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:10.123831 17189 leader_election.cc:304] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:10.123871 17118 raft_consensus.cc:2468] T 800999ad9c144b02b58cf1bf02b6d9e2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 0.
I20260929 02:01:10.124156 17682 raft_consensus.cc:2804] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:10.124271 17682 raft_consensus.cc:493] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:10.124413 17682 raft_consensus.cc:3060] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:10.127053 17682 raft_consensus.cc:515] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.127617 17682 leader_election.cc:290] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:10.128003 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "800999ad9c144b02b58cf1bf02b6d9e2" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:10.128091 17118 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "800999ad9c144b02b58cf1bf02b6d9e2" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:10.128207 16984 raft_consensus.cc:3060] T 800999ad9c144b02b58cf1bf02b6d9e2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:10.128295 17118 raft_consensus.cc:3060] T 800999ad9c144b02b58cf1bf02b6d9e2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:10.130527 16984 raft_consensus.cc:2468] T 800999ad9c144b02b58cf1bf02b6d9e2 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 1.
I20260929 02:01:10.130690 17118 raft_consensus.cc:2468] T 800999ad9c144b02b58cf1bf02b6d9e2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 1.
I20260929 02:01:10.130983 17189 leader_election.cc:304] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:10.131269 17682 raft_consensus.cc:2804] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:10.131443 17682 raft_consensus.cc:697] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Becoming Leader. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Running, Role: LEADER
I20260929 02:01:10.131966 17682 consensus_queue.cc:237] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.134924 16816 catalog_manager.cc:5838] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88 reported cstate change: term changed from 0 to 1, leader changed from <none> to d103dd4823044faebe46100a0dfbee88 (127.0.71.67). New cstate: current_term: 1 leader_uuid: "d103dd4823044faebe46100a0dfbee88" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: HEALTHY } } }
I20260929 02:01:10.143890 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.144239 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.144593 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.145143 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.145460 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.145854 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.148602 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.004s user 0.004s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":173,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:10.149507 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.004s user 0.003s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":173,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:10.277444 17779 consensus_queue.cc:1048] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:10.282523 17781 consensus_queue.cc:1048] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:10.346005 16816 catalog_manager.cc:2546] Servicing SoftDeleteTable request from {username='slave'} at 127.0.0.1:44708:
table { table_name: "table-3" } modify_external_catalogs: true
I20260929 02:01:10.346184 16816 catalog_manager.cc:2794] Servicing DeleteTable request from {username='slave'} at 127.0.0.1:44708:
table { table_name: "table-3" } modify_external_catalogs: true
I20260929 02:01:10.356856 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.358515 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.358946 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.358959 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.359229 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.359251 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.359426 16816 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 5e9b68ceb78f45edaaebaf7314364446
I20260929 02:01:10.359543 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.359536 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.360004 16816 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet d57762935f614c4391a5236fd2cb72c4
I20260929 02:01:10.360296 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.360540 16816 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet c029d8ad8e534719be68303011ec7d8b
I20260929 02:01:10.360586 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.360809 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.360996 16816 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet b8b094aa976542c2bf9646de8377fe18
I20260929 02:01:10.361505 16816 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet e5aa0d7aa540479c89ae9b8b61d89c4e
I20260929 02:01:10.362028 16816 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 5d8486054d464e4ab45c10da4de28090
I20260929 02:01:10.362331 16961 tablet_service.cc:1558] Processing DeleteTablet for tablet 5e9b68ceb78f45edaaebaf7314364446 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:10.362519 16816 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet fed81261ad7d4afabab4ada3e33aa5e2
I20260929 02:01:10.362742 16956 tablet_service.cc:1558] Processing DeleteTablet for tablet d57762935f614c4391a5236fd2cb72c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:10.362917 16962 tablet_service.cc:1558] Processing DeleteTablet for tablet c029d8ad8e534719be68303011ec7d8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:10.362944 16816 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 16931d61d19f4b1eaed27200e43c5336
I20260929 02:01:10.363095 16964 tablet_service.cc:1558] Processing DeleteTablet for tablet b8b094aa976542c2bf9646de8377fe18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:10.363293 16816 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet e050cc7cbfa445e3be6081ca25c9ed63
I20260929 02:01:10.363734 16816 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet e865b8f545e54b7eb0015f6bc1374540
I20260929 02:01:10.364233 16816 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 26ea597431574ea9a145a78687442f50
I20260929 02:01:10.364318 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.007s user 0.005s sys 0.000s Metrics: {"bytes_written":840,"delete_count":0,"lbm_write_time_us":125,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":12}
I20260929 02:01:10.364741 16816 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 2e8c4b957e16467fa6bd6e9c317b1b1f
I20260929 02:01:10.364955 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.811068
I20260929 02:01:10.365252 16955 tablet_service.cc:1558] Processing DeleteTablet for tablet 16931d61d19f4b1eaed27200e43c5336 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:10.365304 16952 tablet_service.cc:1558] Processing DeleteTablet for tablet e865b8f545e54b7eb0015f6bc1374540 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:10.365499 16953 tablet_service.cc:1558] Processing DeleteTablet for tablet e050cc7cbfa445e3be6081ca25c9ed63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:10.365656 16816 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 273fd95488284ef88c2272052f0de976
I20260929 02:01:10.365757 16962 tablet_service.cc:1558] Processing DeleteTablet for tablet 5d8486054d464e4ab45c10da4de28090 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:10.365972 16964 tablet_service.cc:1558] Processing DeleteTablet for tablet e5aa0d7aa540479c89ae9b8b61d89c4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:10.366587 16816 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 800999ad9c144b02b58cf1bf02b6d9e2
I20260929 02:01:10.366645 16956 tablet_service.cc:1558] Processing DeleteTablet for tablet fed81261ad7d4afabab4ada3e33aa5e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:10.366914 17813 tablet_replica.cc:333] T 5e9b68ceb78f45edaaebaf7314364446 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:10.367142 17813 raft_consensus.cc:2243] T 5e9b68ceb78f45edaaebaf7314364446 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.367241 16816 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet f8dd03b8a32f4ec5917415ee176d272b
I20260929 02:01:10.367565 17090 tablet_service.cc:1558] Processing DeleteTablet for tablet c029d8ad8e534719be68303011ec7d8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:10.367839 16816 catalog_manager.cc:6133] T 00000000000000000000000000000000 P 02c2326bb7f24f5c98809afcc5a36d25: Sending DeleteTablet for 3 replicas of tablet 964849b43d28439aa45e8a51084bcae9
I20260929 02:01:10.368055 17097 tablet_service.cc:1558] Processing DeleteTablet for tablet fed81261ad7d4afabab4ada3e33aa5e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:10.368192 17095 tablet_service.cc:1558] Processing DeleteTablet for tablet 16931d61d19f4b1eaed27200e43c5336 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:10.368291 17088 tablet_service.cc:1558] Processing DeleteTablet for tablet e050cc7cbfa445e3be6081ca25c9ed63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:10.368543 17097 tablet_service.cc:1558] Processing DeleteTablet for tablet e865b8f545e54b7eb0015f6bc1374540 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:10.368772 17088 tablet_service.cc:1558] Processing DeleteTablet for tablet 26ea597431574ea9a145a78687442f50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:10.368907 17095 tablet_service.cc:1558] Processing DeleteTablet for tablet 273fd95488284ef88c2272052f0de976 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:10.369011 17088 tablet_service.cc:1558] Processing DeleteTablet for tablet 800999ad9c144b02b58cf1bf02b6d9e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:10.369175 17095 tablet_service.cc:1558] Processing DeleteTablet for tablet f8dd03b8a32f4ec5917415ee176d272b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:10.369251 17097 tablet_service.cc:1558] Processing DeleteTablet for tablet 2e8c4b957e16467fa6bd6e9c317b1b1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:10.369352 17085 tablet_service.cc:1558] Processing DeleteTablet for tablet 5e9b68ceb78f45edaaebaf7314364446 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:10.369508 17088 tablet_service.cc:1558] Processing DeleteTablet for tablet 964849b43d28439aa45e8a51084bcae9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:10.369560 17087 tablet_service.cc:1558] Processing DeleteTablet for tablet d57762935f614c4391a5236fd2cb72c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:10.369657 17228 tablet_service.cc:1558] Processing DeleteTablet for tablet 5e9b68ceb78f45edaaebaf7314364446 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:10.369772 17092 tablet_service.cc:1558] Processing DeleteTablet for tablet b8b094aa976542c2bf9646de8377fe18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:10.370002 17098 tablet_service.cc:1558] Processing DeleteTablet for tablet e5aa0d7aa540479c89ae9b8b61d89c4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:10.370023 17226 tablet_service.cc:1558] Processing DeleteTablet for tablet 5d8486054d464e4ab45c10da4de28090 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:10.370213 17233 tablet_service.cc:1558] Processing DeleteTablet for tablet c029d8ad8e534719be68303011ec7d8b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:10.370232 17096 tablet_service.cc:1558] Processing DeleteTablet for tablet 5d8486054d464e4ab45c10da4de28090 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:50894
I20260929 02:01:10.370462 17229 tablet_service.cc:1558] Processing DeleteTablet for tablet d57762935f614c4391a5236fd2cb72c4 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:10.370584 17218 tablet_service.cc:1558] Processing DeleteTablet for tablet e5aa0d7aa540479c89ae9b8b61d89c4e with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:10.370756 17226 tablet_service.cc:1558] Processing DeleteTablet for tablet fed81261ad7d4afabab4ada3e33aa5e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:10.371073 17219 tablet_service.cc:1558] Processing DeleteTablet for tablet e050cc7cbfa445e3be6081ca25c9ed63 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:10.371238 17233 tablet_service.cc:1558] Processing DeleteTablet for tablet 16931d61d19f4b1eaed27200e43c5336 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:10.370065 17224 tablet_service.cc:1558] Processing DeleteTablet for tablet b8b094aa976542c2bf9646de8377fe18 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:10.371507 17233 tablet_service.cc:1558] Processing DeleteTablet for tablet e865b8f545e54b7eb0015f6bc1374540 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:10.371717 17224 tablet_service.cc:1558] Processing DeleteTablet for tablet 26ea597431574ea9a145a78687442f50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:10.372094 17219 tablet_service.cc:1558] Processing DeleteTablet for tablet 2e8c4b957e16467fa6bd6e9c317b1b1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:10.372606 17219 tablet_service.cc:1558] Processing DeleteTablet for tablet 273fd95488284ef88c2272052f0de976 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:10.373202 17219 tablet_service.cc:1558] Processing DeleteTablet for tablet 800999ad9c144b02b58cf1bf02b6d9e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:10.374070 17219 tablet_service.cc:1558] Processing DeleteTablet for tablet f8dd03b8a32f4ec5917415ee176d272b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:10.374358 16952 tablet_service.cc:1558] Processing DeleteTablet for tablet f8dd03b8a32f4ec5917415ee176d272b with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:10.374521 16953 tablet_service.cc:1558] Processing DeleteTablet for tablet 800999ad9c144b02b58cf1bf02b6d9e2 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:10.374622 17219 tablet_service.cc:1558] Processing DeleteTablet for tablet 964849b43d28439aa45e8a51084bcae9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:53312
I20260929 02:01:10.374804 16955 tablet_service.cc:1558] Processing DeleteTablet for tablet 964849b43d28439aa45e8a51084bcae9 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:10.374996 16962 tablet_service.cc:1558] Processing DeleteTablet for tablet 273fd95488284ef88c2272052f0de976 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:10.375201 16964 tablet_service.cc:1558] Processing DeleteTablet for tablet 2e8c4b957e16467fa6bd6e9c317b1b1f with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:10.375705 16956 tablet_service.cc:1558] Processing DeleteTablet for tablet 26ea597431574ea9a145a78687442f50 with delete_type TABLET_DATA_DELETED (Table deleted at 2026-09-29 02:01:10 UTC) from {username='slave'} at 127.0.0.1:35028
I20260929 02:01:10.376292 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MajorDeltaCompactionOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.367380 17813 raft_consensus.cc:2272] T 5e9b68ceb78f45edaaebaf7314364446 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.378242 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.017s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":16536}
I20260929 02:01:10.378294 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.019s user 0.006s sys 0.000s Metrics: {"bytes_written":555,"delete_count":0,"lbm_write_time_us":182,"lbm_writes_lt_1ms":4,"mutex_wait_us":4776,"reinsert_count":0,"update_count":5}
I20260929 02:01:10.378237 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.018s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":18341}
I20260929 02:01:10.378746 16816 catalog_manager.cc:2296] Servicing CreateTable request from {username='slave'} at 127.0.0.1:44708:
name: "table-4"
schema {
columns {
name: "key"
type: INT64
is_key: true
is_nullable: false
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
columns {
name: "col"
type: INT64
is_key: false
is_nullable: true
encoding: AUTO_ENCODING
compression: DEFAULT_COMPRESSION
cfile_block_size: 0
immutable: false
}
}
num_replicas: 3
split_rows_range_bounds {
}
partition_schema {
hash_schema {
columns {
name: "key"
}
num_buckets: 16
seed: 0
}
}
I20260929 02:01:10.379647 17813 ts_tablet_manager.cc:1962] T 5e9b68ceb78f45edaaebaf7314364446 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 02:01:10.379992 16816 catalog_manager.cc:7226] The number of live tablet servers is not enough to re-replicate a tablet replica of the newly created table table-4 in case of a server failure: 4 tablet servers would be needed, 3 are available. Consider bringing up more tablet servers.
I20260929 02:01:10.383733 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:10.383972 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.377002 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.017s user 0.000s sys 0.005s Metrics: {"bytes_written":969,"delete_count":0,"lbm_write_time_us":174,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":17}
I20260929 02:01:10.388772 17814 tablet_replica.cc:333] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:10.389021 17814 raft_consensus.cc:2243] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:10.389426 17814 raft_consensus.cc:2272] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.390266 17814 ts_tablet_manager.cc:1962] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.392204 17813 ts_tablet_manager.cc:1975] T 5e9b68ceb78f45edaaebaf7314364446 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.392402 17813 log.cc:1199] T 5e9b68ceb78f45edaaebaf7314364446 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/5e9b68ceb78f45edaaebaf7314364446
I20260929 02:01:10.393016 17813 ts_tablet_manager.cc:1996] T 5e9b68ceb78f45edaaebaf7314364446 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:10.394060 17813 tablet_replica.cc:333] T d57762935f614c4391a5236fd2cb72c4 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:10.394339 17813 raft_consensus.cc:2243] T d57762935f614c4391a5236fd2cb72c4 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.394564 17813 raft_consensus.cc:2272] T d57762935f614c4391a5236fd2cb72c4 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.394731 17815 tablet_replica.cc:333] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:10.394973 17815 raft_consensus.cc:2243] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:10.395407 17815 raft_consensus.cc:2272] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.395619 17813 ts_tablet_manager.cc:1962] T d57762935f614c4391a5236fd2cb72c4 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.396116 17815 ts_tablet_manager.cc:1962] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.396514 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 5e9b68ceb78f45edaaebaf7314364446 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.398751 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.399988 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushMRSOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.403414 17813 ts_tablet_manager.cc:1975] T d57762935f614c4391a5236fd2cb72c4 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.404389 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.025s user 0.004s sys 0.005s Metrics: {"cfile_cache_miss":7,"cfile_cache_miss_bytes":6489,"cfile_init":5,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":68,"lbm_read_time_us":560,"lbm_reads_lt_1ms":27,"lbm_write_time_us":341,"lbm_writes_lt_1ms":9,"peak_mem_usage":32768,"reinsert_count":0,"update_count":18}
I20260929 02:01:10.404639 17813 log.cc:1199] T d57762935f614c4391a5236fd2cb72c4 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/d57762935f614c4391a5236fd2cb72c4
I20260929 02:01:10.405278 17813 ts_tablet_manager.cc:1996] T d57762935f614c4391a5236fd2cb72c4 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:10.406558 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet d57762935f614c4391a5236fd2cb72c4 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.407053 17813 tablet_replica.cc:333] T c029d8ad8e534719be68303011ec7d8b P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:10.407305 17813 raft_consensus.cc:2243] T c029d8ad8e534719be68303011ec7d8b P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.407663 17813 raft_consensus.cc:2272] T c029d8ad8e534719be68303011ec7d8b P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.408689 17813 ts_tablet_manager.cc:1962] T c029d8ad8e534719be68303011ec7d8b P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.411336 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MajorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.046s user 0.013s sys 0.000s Metrics: {"cfile_cache_miss":9,"cfile_cache_miss_bytes":6814,"cfile_init":6,"delete_count":0,"delta_blocks_compacted":3,"delta_iterators_relevant":3,"dirs.queue_time_us":3697,"dirs.run_cpu_time_us":126,"dirs.run_wall_time_us":606,"lbm_read_time_us":747,"lbm_reads_lt_1ms":33,"lbm_write_time_us":315,"lbm_writes_lt_1ms":9,"mutex_wait_us":16970,"peak_mem_usage":32768,"reinsert_count":0,"update_count":18}
I20260929 02:01:10.413704 17815 ts_tablet_manager.cc:1975] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.413849 17815 log.cc:1199] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/5e9b68ceb78f45edaaebaf7314364446
I20260929 02:01:10.414314 17815 ts_tablet_manager.cc:1996] T 5e9b68ceb78f45edaaebaf7314364446 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:10.416030 17815 tablet_replica.cc:333] T 5d8486054d464e4ab45c10da4de28090 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:10.416146 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 5e9b68ceb78f45edaaebaf7314364446 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.416265 17815 raft_consensus.cc:2243] T 5d8486054d464e4ab45c10da4de28090 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.416465 17815 raft_consensus.cc:2272] T 5d8486054d464e4ab45c10da4de28090 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.416483 17813 ts_tablet_manager.cc:1975] T c029d8ad8e534719be68303011ec7d8b P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.416548 17814 ts_tablet_manager.cc:1975] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.416688 17814 log.cc:1199] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/c029d8ad8e534719be68303011ec7d8b
I20260929 02:01:10.417074 17813 log.cc:1199] T c029d8ad8e534719be68303011ec7d8b P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/c029d8ad8e534719be68303011ec7d8b
I20260929 02:01:10.417192 17814 ts_tablet_manager.cc:1996] T c029d8ad8e534719be68303011ec7d8b P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:10.417629 17813 ts_tablet_manager.cc:1996] T c029d8ad8e534719be68303011ec7d8b P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:10.417869 17815 ts_tablet_manager.cc:1962] T 5d8486054d464e4ab45c10da4de28090 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.418277 17814 tablet_replica.cc:333] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:10.418419 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet c029d8ad8e534719be68303011ec7d8b (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.418509 17814 raft_consensus.cc:2243] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:10.418901 17814 raft_consensus.cc:2272] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.418953 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet c029d8ad8e534719be68303011ec7d8b (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.419866 17813 tablet_replica.cc:333] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:10.420235 17813 raft_consensus.cc:2243] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:10.420635 17813 raft_consensus.cc:2272] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.421594 17813 ts_tablet_manager.cc:1962] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.421729 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.023s user 0.011s sys 0.003s Metrics: {"bytes_written":6940,"cfile_init":1,"dirs.queue_time_us":205,"dirs.run_cpu_time_us":105,"dirs.run_wall_time_us":599,"drs_written":1,"lbm_read_time_us":109,"lbm_reads_lt_1ms":4,"lbm_write_time_us":978,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":17}
I20260929 02:01:10.422818 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260929 02:01:10.423134 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260929 02:01:10.423429 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260929 02:01:10.423291 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushMRSOp(00000000000000000000000000000000) complete. Timing: real 0.023s user 0.010s sys 0.000s Metrics: {"bytes_written":6940,"cfile_init":1,"dirs.queue_time_us":40,"dirs.run_cpu_time_us":178,"dirs.run_wall_time_us":1065,"drs_written":1,"lbm_read_time_us":69,"lbm_reads_lt_1ms":4,"lbm_write_time_us":844,"lbm_writes_lt_1ms":27,"peak_mem_usage":0,"rows_written":17}
I20260929 02:01:10.423678 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=1.936300
I20260929 02:01:10.424268 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":92,"lbm_reads_lt_1ms":4}
I20260929 02:01:10.425344 17815 ts_tablet_manager.cc:1975] T 5d8486054d464e4ab45c10da4de28090 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.425506 17815 log.cc:1199] T 5d8486054d464e4ab45c10da4de28090 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/5d8486054d464e4ab45c10da4de28090
I20260929 02:01:10.425572 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.425937 17815 ts_tablet_manager.cc:1996] T 5d8486054d464e4ab45c10da4de28090 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:10.427017 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 5d8486054d464e4ab45c10da4de28090 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.427379 17815 tablet_replica.cc:333] T c029d8ad8e534719be68303011ec7d8b P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:10.425119 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.427560 17815 raft_consensus.cc:2243] T c029d8ad8e534719be68303011ec7d8b P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.428774 17815 raft_consensus.cc:2272] T c029d8ad8e534719be68303011ec7d8b P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.429421 17813 ts_tablet_manager.cc:1975] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.429615 17813 log.cc:1199] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/b8b094aa976542c2bf9646de8377fe18
I20260929 02:01:10.430017 17815 ts_tablet_manager.cc:1962] T c029d8ad8e534719be68303011ec7d8b P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.430182 17813 ts_tablet_manager.cc:1996] T b8b094aa976542c2bf9646de8377fe18 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:10.431281 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet b8b094aa976542c2bf9646de8377fe18 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.431694 17813 tablet_replica.cc:333] T 16931d61d19f4b1eaed27200e43c5336 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:10.431977 17813 raft_consensus.cc:2243] T 16931d61d19f4b1eaed27200e43c5336 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.432241 17813 raft_consensus.cc:2272] T 16931d61d19f4b1eaed27200e43c5336 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.432761 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 249 bytes on disk
I20260929 02:01:10.433135 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.433755 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":94,"lbm_reads_lt_1ms":4}
I20260929 02:01:10.435256 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.436021 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling CompactRowSetsOp(00000000000000000000000000000000): perf score=1.100543
I20260929 02:01:10.436276 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.436220 17815 ts_tablet_manager.cc:1975] T c029d8ad8e534719be68303011ec7d8b P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.436501 17815 log.cc:1199] T c029d8ad8e534719be68303011ec7d8b P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/c029d8ad8e534719be68303011ec7d8b
I20260929 02:01:10.436975 17815 ts_tablet_manager.cc:1996] T c029d8ad8e534719be68303011ec7d8b P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:10.437922 17815 tablet_replica.cc:333] T d57762935f614c4391a5236fd2cb72c4 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:10.438135 17815 raft_consensus.cc:2243] T d57762935f614c4391a5236fd2cb72c4 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.438627 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet c029d8ad8e534719be68303011ec7d8b (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.438658 17781 consensus_queue.cc:1048] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:10.438872 17815 raft_consensus.cc:2272] T d57762935f614c4391a5236fd2cb72c4 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.439646 17815 ts_tablet_manager.cc:1962] T d57762935f614c4391a5236fd2cb72c4 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.439873 17814 ts_tablet_manager.cc:1962] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.446519 17815 ts_tablet_manager.cc:1975] T d57762935f614c4391a5236fd2cb72c4 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.446692 17815 log.cc:1199] T d57762935f614c4391a5236fd2cb72c4 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/d57762935f614c4391a5236fd2cb72c4
I20260929 02:01:10.447124 17815 ts_tablet_manager.cc:1996] T d57762935f614c4391a5236fd2cb72c4 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:10.448276 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet d57762935f614c4391a5236fd2cb72c4 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.449931 17815 tablet_replica.cc:333] T e5aa0d7aa540479c89ae9b8b61d89c4e P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:10.450111 17815 raft_consensus.cc:2243] T e5aa0d7aa540479c89ae9b8b61d89c4e P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.450343 17815 raft_consensus.cc:2272] T e5aa0d7aa540479c89ae9b8b61d89c4e P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.452322 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.453613 17814 ts_tablet_manager.cc:1975] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.453707 17815 ts_tablet_manager.cc:1962] T e5aa0d7aa540479c89ae9b8b61d89c4e P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.455263 17814 log.cc:1199] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/fed81261ad7d4afabab4ada3e33aa5e2
I20260929 02:01:10.455569 17779 consensus_queue.cc:1048] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:10.455852 17813 ts_tablet_manager.cc:1962] T 16931d61d19f4b1eaed27200e43c5336 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.459807 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.026s user 0.005s sys 0.000s Metrics: {"bytes_written":597,"delete_count":0,"lbm_write_time_us":157,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":11}
I20260929 02:01:10.460222 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.460613 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.460899 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.462271 17813 ts_tablet_manager.cc:1975] T 16931d61d19f4b1eaed27200e43c5336 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.462393 17813 log.cc:1199] T 16931d61d19f4b1eaed27200e43c5336 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/16931d61d19f4b1eaed27200e43c5336
I20260929 02:01:10.462327 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.462606 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.463399 17814 ts_tablet_manager.cc:1996] T fed81261ad7d4afabab4ada3e33aa5e2 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:10.464551 17814 tablet_replica.cc:333] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:10.464372 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.012s user 0.004s sys 0.000s Metrics: {"bytes_written":497,"delete_count":0,"lbm_write_time_us":164,"lbm_writes_lt_1ms":4,"mutex_wait_us":5468,"reinsert_count":0,"update_count":5}
I20260929 02:01:10.464828 17814 raft_consensus.cc:2243] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:10.465027 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet fed81261ad7d4afabab4ada3e33aa5e2 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.465279 17814 raft_consensus.cc:2272] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.466979 16961 tablet_service.cc:1511] Processing CreateTablet for tablet 61923b7b8662455e8b7f8d394dfbf021 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.467381 16961 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61923b7b8662455e8b7f8d394dfbf021. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.468595 17090 tablet_service.cc:1511] Processing CreateTablet for tablet 61923b7b8662455e8b7f8d394dfbf021 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.469004 17090 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61923b7b8662455e8b7f8d394dfbf021. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.468999 17228 tablet_service.cc:1511] Processing CreateTablet for tablet 61923b7b8662455e8b7f8d394dfbf021 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 0, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.469414 17228 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 61923b7b8662455e8b7f8d394dfbf021. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.470599 16964 tablet_service.cc:1511] Processing CreateTablet for tablet 3843521ef989409da22fdb563bb50689 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.471643 16956 tablet_service.cc:1511] Processing CreateTablet for tablet 942ff8f711b849e1a89a807e869354f1 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.472189 16964 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3843521ef989409da22fdb563bb50689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.474967 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.475294 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.475773 16955 tablet_service.cc:1511] Processing CreateTablet for tablet a3e7696ec7984a8594f0b10fd53cbf71 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.476133 16955 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3e7696ec7984a8594f0b10fd53cbf71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.476702 16953 tablet_service.cc:1511] Processing CreateTablet for tablet 1c5034ef0b384105a1d97e01d9fead7d (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.477092 16953 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c5034ef0b384105a1d97e01d9fead7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.481135 16962 tablet_service.cc:1511] Processing CreateTablet for tablet 6037cdfdeaaf4ee29081f8f7d849bf1e (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.481534 16952 tablet_service.cc:1511] Processing CreateTablet for tablet 5663bfc5d2d34aa4b097e6d05e176d16 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.481671 16962 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6037cdfdeaaf4ee29081f8f7d849bf1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.481926 16952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5663bfc5d2d34aa4b097e6d05e176d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.482025 16963 tablet_service.cc:1511] Processing CreateTablet for tablet 10281ddd7db046fa8f4e275082630dca (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.482254 16956 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 942ff8f711b849e1a89a807e869354f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.482594 16954 tablet_service.cc:1511] Processing CreateTablet for tablet 8f12f916b4ee494f94c2007983a440d2 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.482970 16954 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f12f916b4ee494f94c2007983a440d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.483273 16963 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10281ddd7db046fa8f4e275082630dca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.483567 16951 tablet_service.cc:1511] Processing CreateTablet for tablet aed0f67930b64ac78c719794f3a3198e (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.483942 16951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aed0f67930b64ac78c719794f3a3198e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.484340 17815 ts_tablet_manager.cc:1975] T e5aa0d7aa540479c89ae9b8b61d89c4e P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.484508 17815 log.cc:1199] T e5aa0d7aa540479c89ae9b8b61d89c4e P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/e5aa0d7aa540479c89ae9b8b61d89c4e
I20260929 02:01:10.484980 17815 ts_tablet_manager.cc:1996] T e5aa0d7aa540479c89ae9b8b61d89c4e P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:10.486073 17815 tablet_replica.cc:333] T fed81261ad7d4afabab4ada3e33aa5e2 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:10.486279 17815 raft_consensus.cc:2243] T fed81261ad7d4afabab4ada3e33aa5e2 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.486439 17815 raft_consensus.cc:2272] T fed81261ad7d4afabab4ada3e33aa5e2 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.487341 17815 ts_tablet_manager.cc:1962] T fed81261ad7d4afabab4ada3e33aa5e2 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.488775 16965 tablet_service.cc:1511] Processing CreateTablet for tablet 3990f27c54a54031a7f554b9633f56f4 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.489203 16965 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3990f27c54a54031a7f554b9633f56f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.491827 17781 consensus_queue.cc:1048] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:10.493657 17224 tablet_service.cc:1511] Processing CreateTablet for tablet 3843521ef989409da22fdb563bb50689 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.494030 17224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3843521ef989409da22fdb563bb50689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.494360 17219 tablet_service.cc:1511] Processing CreateTablet for tablet 942ff8f711b849e1a89a807e869354f1 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.494477 17815 ts_tablet_manager.cc:1975] T fed81261ad7d4afabab4ada3e33aa5e2 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.494640 17815 log.cc:1199] T fed81261ad7d4afabab4ada3e33aa5e2 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/fed81261ad7d4afabab4ada3e33aa5e2
I20260929 02:01:10.494765 17219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 942ff8f711b849e1a89a807e869354f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.495100 17815 ts_tablet_manager.cc:1996] T fed81261ad7d4afabab4ada3e33aa5e2 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:10.496093 17815 tablet_replica.cc:333] T e050cc7cbfa445e3be6081ca25c9ed63 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:10.496290 17815 raft_consensus.cc:2243] T e050cc7cbfa445e3be6081ca25c9ed63 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.496533 17815 raft_consensus.cc:2272] T e050cc7cbfa445e3be6081ca25c9ed63 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.496778 17226 tablet_service.cc:1511] Processing CreateTablet for tablet a3e7696ec7984a8594f0b10fd53cbf71 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.496771 17233 tablet_service.cc:1511] Processing CreateTablet for tablet 6037cdfdeaaf4ee29081f8f7d849bf1e (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.497167 17226 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3e7696ec7984a8594f0b10fd53cbf71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.497435 17233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6037cdfdeaaf4ee29081f8f7d849bf1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.497529 17815 ts_tablet_manager.cc:1962] T e050cc7cbfa445e3be6081ca25c9ed63 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.497877 17218 tablet_service.cc:1511] Processing CreateTablet for tablet 1c5034ef0b384105a1d97e01d9fead7d (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.498238 17218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c5034ef0b384105a1d97e01d9fead7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.499577 17229 tablet_service.cc:1511] Processing CreateTablet for tablet 5663bfc5d2d34aa4b097e6d05e176d16 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.499965 17229 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5663bfc5d2d34aa4b097e6d05e176d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.499898 17225 tablet_service.cc:1511] Processing CreateTablet for tablet 8f12f916b4ee494f94c2007983a440d2 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.500248 17225 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f12f916b4ee494f94c2007983a440d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.505300 17815 ts_tablet_manager.cc:1975] T e050cc7cbfa445e3be6081ca25c9ed63 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.505452 17815 log.cc:1199] T e050cc7cbfa445e3be6081ca25c9ed63 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/e050cc7cbfa445e3be6081ca25c9ed63
I20260929 02:01:10.505935 17815 ts_tablet_manager.cc:1996] T e050cc7cbfa445e3be6081ca25c9ed63 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:10.506824 17815 tablet_replica.cc:333] T 16931d61d19f4b1eaed27200e43c5336 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:10.507021 17815 raft_consensus.cc:2243] T 16931d61d19f4b1eaed27200e43c5336 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.507179 17815 raft_consensus.cc:2272] T 16931d61d19f4b1eaed27200e43c5336 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.507951 17814 ts_tablet_manager.cc:1962] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.508148 17815 ts_tablet_manager.cc:1962] T 16931d61d19f4b1eaed27200e43c5336 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.508981 17223 tablet_service.cc:1511] Processing CreateTablet for tablet aed0f67930b64ac78c719794f3a3198e (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.509388 17223 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aed0f67930b64ac78c719794f3a3198e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.515374 17815 ts_tablet_manager.cc:1975] T 16931d61d19f4b1eaed27200e43c5336 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.515542 17815 log.cc:1199] T 16931d61d19f4b1eaed27200e43c5336 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/16931d61d19f4b1eaed27200e43c5336
I20260929 02:01:10.516023 17815 ts_tablet_manager.cc:1996] T 16931d61d19f4b1eaed27200e43c5336 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:10.516666 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet e5aa0d7aa540479c89ae9b8b61d89c4e (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.517267 17815 tablet_replica.cc:333] T b8b094aa976542c2bf9646de8377fe18 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:10.517447 17815 raft_consensus.cc:2243] T b8b094aa976542c2bf9646de8377fe18 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.517633 17815 raft_consensus.cc:2272] T b8b094aa976542c2bf9646de8377fe18 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.518535 17815 ts_tablet_manager.cc:1962] T b8b094aa976542c2bf9646de8377fe18 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.526573 17815 ts_tablet_manager.cc:1975] T b8b094aa976542c2bf9646de8377fe18 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.526688 17815 log.cc:1199] T b8b094aa976542c2bf9646de8377fe18 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/b8b094aa976542c2bf9646de8377fe18
I20260929 02:01:10.527110 17815 ts_tablet_manager.cc:1996] T b8b094aa976542c2bf9646de8377fe18 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:10.527091 17814 ts_tablet_manager.cc:1975] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.527205 17814 log.cc:1199] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/16931d61d19f4b1eaed27200e43c5336
I20260929 02:01:10.527621 17814 ts_tablet_manager.cc:1996] T 16931d61d19f4b1eaed27200e43c5336 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:10.528055 17815 tablet_replica.cc:333] T e865b8f545e54b7eb0015f6bc1374540 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:10.528246 17815 raft_consensus.cc:2243] T e865b8f545e54b7eb0015f6bc1374540 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.528395 17815 raft_consensus.cc:2272] T e865b8f545e54b7eb0015f6bc1374540 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.529036 17815 ts_tablet_manager.cc:1962] T e865b8f545e54b7eb0015f6bc1374540 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.532961 17221 tablet_service.cc:1511] Processing CreateTablet for tablet 3990f27c54a54031a7f554b9633f56f4 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.533847 17221 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3990f27c54a54031a7f554b9633f56f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.535009 17682 consensus_queue.cc:1048] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:10.462852 17813 ts_tablet_manager.cc:1996] T 16931d61d19f4b1eaed27200e43c5336 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:10.537379 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet fed81261ad7d4afabab4ada3e33aa5e2 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.537602 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet e050cc7cbfa445e3be6081ca25c9ed63 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.537771 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 16931d61d19f4b1eaed27200e43c5336 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.537906 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet b8b094aa976542c2bf9646de8377fe18 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.538477 17779 consensus_queue.cc:1048] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f [LEADER]: Connected to new peer: Peer: permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:10.539508 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 16931d61d19f4b1eaed27200e43c5336 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.540030 17815 ts_tablet_manager.cc:1975] T e865b8f545e54b7eb0015f6bc1374540 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 0.0
I20260929 02:01:10.540200 17815 log.cc:1199] T e865b8f545e54b7eb0015f6bc1374540 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/e865b8f545e54b7eb0015f6bc1374540
I20260929 02:01:10.540652 17815 ts_tablet_manager.cc:1996] T e865b8f545e54b7eb0015f6bc1374540 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:10.538448 17813 tablet_replica.cc:333] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:10.541628 17815 tablet_replica.cc:333] T 26ea597431574ea9a145a78687442f50 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:10.541850 17815 raft_consensus.cc:2243] T 26ea597431574ea9a145a78687442f50 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.542084 17815 raft_consensus.cc:2272] T 26ea597431574ea9a145a78687442f50 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.542150 17813 raft_consensus.cc:2243] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:10.542654 17813 pending_rounds.cc:70] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f: Trying to abort 1 pending ops.
I20260929 02:01:10.543357 17813 pending_rounds.cc:77] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7334491217414877184 op_type: NO_OP noop_request { }
I20260929 02:01:10.544250 17813 raft_consensus.cc:2889] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: NO_OP replication failed: Aborted: Op aborted
I20260929 02:01:10.544643 17813 raft_consensus.cc:2272] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.545740 17680 consensus_queue.cc:1048] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88 [LEADER]: Connected to new peer: Peer: permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 }, Status: LMP_MISMATCH, Last received: 0.0, Next index: 1, Last known committed idx: 0, Time since last communication: 0.000s
I20260929 02:01:10.547287 17813 ts_tablet_manager.cc:1962] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.548177 17815 ts_tablet_manager.cc:1962] T 26ea597431574ea9a145a78687442f50 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.549163 16951 tablet_service.cc:1511] Processing CreateTablet for tablet e15244567c6a4104a45c3d118eeab335 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.549629 16951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e15244567c6a4104a45c3d118eeab335. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.550771 17814 tablet_replica.cc:333] T e050cc7cbfa445e3be6081ca25c9ed63 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:10.553093 17814 raft_consensus.cc:2243] T e050cc7cbfa445e3be6081ca25c9ed63 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.554723 16965 tablet_service.cc:1511] Processing CreateTablet for tablet 4ff036da7c4b4975a2f790b6a80ce050 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.555168 16965 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ff036da7c4b4975a2f790b6a80ce050. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.555701 17814 raft_consensus.cc:2272] T e050cc7cbfa445e3be6081ca25c9ed63 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.556609 17815 ts_tablet_manager.cc:1975] T 26ea597431574ea9a145a78687442f50 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.556741 17815 log.cc:1199] T 26ea597431574ea9a145a78687442f50 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/26ea597431574ea9a145a78687442f50
I20260929 02:01:10.557220 17815 ts_tablet_manager.cc:1996] T 26ea597431574ea9a145a78687442f50 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:10.558215 17815 tablet_replica.cc:333] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:10.558439 17815 raft_consensus.cc:2243] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [term 2 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.558640 17815 raft_consensus.cc:2272] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88 [term 2 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.559362 17815 ts_tablet_manager.cc:1962] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.558491 17814 ts_tablet_manager.cc:1962] T e050cc7cbfa445e3be6081ca25c9ed63 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
W20260929 02:01:10.562403 16921 consensus_peers.cc:597] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f -> Peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Couldn't send request to peer d103dd4823044faebe46100a0dfbee88. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: e5aa0d7aa540479c89ae9b8b61d89c4e. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:01:10.562871 17224 tablet_service.cc:1511] Processing CreateTablet for tablet e15244567c6a4104a45c3d118eeab335 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.563401 17224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e15244567c6a4104a45c3d118eeab335. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.563908 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet e865b8f545e54b7eb0015f6bc1374540 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.564366 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 26ea597431574ea9a145a78687442f50 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.565340 17233 tablet_service.cc:1511] Processing CreateTablet for tablet 4ff036da7c4b4975a2f790b6a80ce050 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.565778 17233 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ff036da7c4b4975a2f790b6a80ce050. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.566439 17815 ts_tablet_manager.cc:1975] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20260929 02:01:10.566581 17815 log.cc:1199] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/2e8c4b957e16467fa6bd6e9c317b1b1f
I20260929 02:01:10.567037 17815 ts_tablet_manager.cc:1996] T 2e8c4b957e16467fa6bd6e9c317b1b1f P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:10.568212 17815 tablet_replica.cc:333] T 273fd95488284ef88c2272052f0de976 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:10.568391 17815 raft_consensus.cc:2243] T 273fd95488284ef88c2272052f0de976 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.568686 17815 raft_consensus.cc:2272] T 273fd95488284ef88c2272052f0de976 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.569816 17815 ts_tablet_manager.cc:1962] T 273fd95488284ef88c2272052f0de976 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.570101 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 16931d61d19f4b1eaed27200e43c5336 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.571206 17092 tablet_service.cc:1511] Processing CreateTablet for tablet 6037cdfdeaaf4ee29081f8f7d849bf1e (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 3, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.571653 17092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 6037cdfdeaaf4ee29081f8f7d849bf1e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.571038 17096 tablet_service.cc:1511] Processing CreateTablet for tablet 942ff8f711b849e1a89a807e869354f1 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 1, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.572050 17096 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 942ff8f711b849e1a89a807e869354f1. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.576180 17819 tablet_bootstrap.cc:492] T 3843521ef989409da22fdb563bb50689 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:10.576282 17815 ts_tablet_manager.cc:1975] T 273fd95488284ef88c2272052f0de976 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.576381 17815 log.cc:1199] T 273fd95488284ef88c2272052f0de976 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/273fd95488284ef88c2272052f0de976
I20260929 02:01:10.571575 17098 tablet_service.cc:1511] Processing CreateTablet for tablet 3843521ef989409da22fdb563bb50689 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 2, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.576867 17098 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3843521ef989409da22fdb563bb50689. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.576788 17087 tablet_service.cc:1511] Processing CreateTablet for tablet a3e7696ec7984a8594f0b10fd53cbf71 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 4, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.576867 17815 ts_tablet_manager.cc:1996] T 273fd95488284ef88c2272052f0de976 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:10.577617 17087 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet a3e7696ec7984a8594f0b10fd53cbf71. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.578059 17088 tablet_service.cc:1511] Processing CreateTablet for tablet 1c5034ef0b384105a1d97e01d9fead7d (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 5, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.578323 17097 tablet_service.cc:1511] Processing CreateTablet for tablet 8f12f916b4ee494f94c2007983a440d2 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 7, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.578420 17088 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 1c5034ef0b384105a1d97e01d9fead7d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.577746 17815 tablet_replica.cc:333] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:10.578775 17097 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 8f12f916b4ee494f94c2007983a440d2. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.579346 17815 raft_consensus.cc:2243] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:10.579517 17819 tablet_bootstrap.cc:654] T 3843521ef989409da22fdb563bb50689 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.579751 17815 raft_consensus.cc:2272] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.579743 17085 tablet_service.cc:1511] Processing CreateTablet for tablet 5663bfc5d2d34aa4b097e6d05e176d16 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 6, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.580119 17085 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 5663bfc5d2d34aa4b097e6d05e176d16. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.580729 17815 ts_tablet_manager.cc:1962] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.580943 17095 tablet_service.cc:1511] Processing CreateTablet for tablet aed0f67930b64ac78c719794f3a3198e (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 8, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.581068 17091 tablet_service.cc:1511] Processing CreateTablet for tablet 3990f27c54a54031a7f554b9633f56f4 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 9, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.581316 17095 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet aed0f67930b64ac78c719794f3a3198e. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.581509 17091 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 3990f27c54a54031a7f554b9633f56f4. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.584836 17100 tablet_service.cc:1511] Processing CreateTablet for tablet 10281ddd7db046fa8f4e275082630dca (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.585405 17094 tablet_service.cc:1511] Processing CreateTablet for tablet e15244567c6a4104a45c3d118eeab335 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 11, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.585666 17100 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10281ddd7db046fa8f4e275082630dca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.585911 17094 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet e15244567c6a4104a45c3d118eeab335. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.587132 17815 ts_tablet_manager.cc:1975] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.587256 17815 log.cc:1199] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/800999ad9c144b02b58cf1bf02b6d9e2
I20260929 02:01:10.587630 17815 ts_tablet_manager.cc:1996] T 800999ad9c144b02b58cf1bf02b6d9e2 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:10.588600 17815 tablet_replica.cc:333] T f8dd03b8a32f4ec5917415ee176d272b P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:10.588760 17815 raft_consensus.cc:2243] T f8dd03b8a32f4ec5917415ee176d272b P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.588939 17815 raft_consensus.cc:2272] T f8dd03b8a32f4ec5917415ee176d272b P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.589805 17815 ts_tablet_manager.cc:1962] T f8dd03b8a32f4ec5917415ee176d272b P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.592828 17099 tablet_service.cc:1511] Processing CreateTablet for tablet 4ff036da7c4b4975a2f790b6a80ce050 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 12, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.593235 17099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 4ff036da7c4b4975a2f790b6a80ce050. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.596221 17815 ts_tablet_manager.cc:1975] T f8dd03b8a32f4ec5917415ee176d272b P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.596330 17815 log.cc:1199] T f8dd03b8a32f4ec5917415ee176d272b P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/f8dd03b8a32f4ec5917415ee176d272b
I20260929 02:01:10.596793 17815 ts_tablet_manager.cc:1996] T f8dd03b8a32f4ec5917415ee176d272b P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:10.597683 17815 tablet_replica.cc:333] T 964849b43d28439aa45e8a51084bcae9 P d103dd4823044faebe46100a0dfbee88: stopping tablet replica
I20260929 02:01:10.597865 17815 raft_consensus.cc:2243] T 964849b43d28439aa45e8a51084bcae9 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.598053 17815 raft_consensus.cc:2272] T 964849b43d28439aa45e8a51084bcae9 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.599030 17815 ts_tablet_manager.cc:1962] T 964849b43d28439aa45e8a51084bcae9 P d103dd4823044faebe46100a0dfbee88: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.577847 17820 tablet_bootstrap.cc:492] T 1c5034ef0b384105a1d97e01d9fead7d P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:10.604107 16964 tablet_service.cc:1511] Processing CreateTablet for tablet ce78e617719b475991ff07a05136c47d (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.604646 16964 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce78e617719b475991ff07a05136c47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.605816 17815 ts_tablet_manager.cc:1975] T 964849b43d28439aa45e8a51084bcae9 P d103dd4823044faebe46100a0dfbee88: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.605960 17815 log.cc:1199] T 964849b43d28439aa45e8a51084bcae9 P d103dd4823044faebe46100a0dfbee88: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-2/wal/wals/964849b43d28439aa45e8a51084bcae9
I20260929 02:01:10.606388 17815 ts_tablet_manager.cc:1996] T 964849b43d28439aa45e8a51084bcae9 P d103dd4823044faebe46100a0dfbee88: Deleting consensus metadata
I20260929 02:01:10.608844 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 2e8c4b957e16467fa6bd6e9c317b1b1f (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.609292 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 273fd95488284ef88c2272052f0de976 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.609607 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 800999ad9c144b02b58cf1bf02b6d9e2 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.609817 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet f8dd03b8a32f4ec5917415ee176d272b (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.610141 16802 catalog_manager.cc:5063] TS d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): tablet 964849b43d28439aa45e8a51084bcae9 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.610565 16952 tablet_service.cc:1511] Processing CreateTablet for tablet b338603394cb4ee087498b8d63b7ded5 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.612387 16952 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b338603394cb4ee087498b8d63b7ded5. 1 dirs total, 0 dirs full, 0 dirs failed
W20260929 02:01:10.619966 17056 consensus_peers.cc:597] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc -> Peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Couldn't send request to peer 97df55a0361841319adfae27c9787b2f. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: d57762935f614c4391a5236fd2cb72c4. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:01:10.623751 17814 ts_tablet_manager.cc:1975] T e050cc7cbfa445e3be6081ca25c9ed63 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.623951 17820 tablet_bootstrap.cc:654] T 1c5034ef0b384105a1d97e01d9fead7d P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.626808 16951 tablet_service.cc:1511] Processing CreateTablet for tablet 93799d6bf3c8407589d9cfb48dd1abd7 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.625921 17218 tablet_service.cc:1511] Processing CreateTablet for tablet b338603394cb4ee087498b8d63b7ded5 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.627291 16951 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93799d6bf3c8407589d9cfb48dd1abd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.627693 17218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b338603394cb4ee087498b8d63b7ded5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.624687 17092 tablet_service.cc:1511] Processing CreateTablet for tablet ce78e617719b475991ff07a05136c47d (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.629182 17092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce78e617719b475991ff07a05136c47d. 1 dirs total, 0 dirs full, 0 dirs failed
W20260929 02:01:10.631139 16921 consensus_peers.cc:597] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f -> Peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Couldn't send request to peer d103dd4823044faebe46100a0dfbee88. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 2e8c4b957e16467fa6bd6e9c317b1b1f. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:01:10.636179 17814 log.cc:1199] T e050cc7cbfa445e3be6081ca25c9ed63 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/e050cc7cbfa445e3be6081ca25c9ed63
I20260929 02:01:10.636678 17219 tablet_service.cc:1511] Processing CreateTablet for tablet 10281ddd7db046fa8f4e275082630dca (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 10, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.636891 17814 ts_tablet_manager.cc:1996] T e050cc7cbfa445e3be6081ca25c9ed63 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:10.638295 17814 tablet_replica.cc:333] T e865b8f545e54b7eb0015f6bc1374540 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:10.639046 17218 tablet_service.cc:1511] Processing CreateTablet for tablet 93799d6bf3c8407589d9cfb48dd1abd7 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.639359 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet e050cc7cbfa445e3be6081ca25c9ed63 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.639101 17099 tablet_service.cc:1511] Processing CreateTablet for tablet 93799d6bf3c8407589d9cfb48dd1abd7 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 15, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.640746 17099 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93799d6bf3c8407589d9cfb48dd1abd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.640867 17813 ts_tablet_manager.cc:1975] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.641396 17813 log.cc:1199] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/e865b8f545e54b7eb0015f6bc1374540
I20260929 02:01:10.642630 17813 ts_tablet_manager.cc:1996] T e865b8f545e54b7eb0015f6bc1374540 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:10.642097 17819 tablet_bootstrap.cc:492] T 3843521ef989409da22fdb563bb50689 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:10.642937 17819 ts_tablet_manager.cc:1449] T 3843521ef989409da22fdb563bb50689 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.067s user 0.004s sys 0.000s
I20260929 02:01:10.643805 17819 raft_consensus.cc:359] T 3843521ef989409da22fdb563bb50689 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.644142 17819 raft_consensus.cc:385] T 3843521ef989409da22fdb563bb50689 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:10.644366 17819 raft_consensus.cc:740] T 3843521ef989409da22fdb563bb50689 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:10.644734 17819 consensus_queue.cc:260] T 3843521ef989409da22fdb563bb50689 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.646132 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.182s user 0.004s sys 0.000s Metrics: {"bytes_written":651,"delete_count":0,"lbm_write_time_us":137,"lbm_writes_lt_1ms":4,"reinsert_count":0,"spinlock_wait_cycles":108958464,"update_count":14}
I20260929 02:01:10.646310 17819 ts_tablet_manager.cc:1480] T 3843521ef989409da22fdb563bb50689 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.003s user 0.000s sys 0.002s
I20260929 02:01:10.646577 17819 tablet_bootstrap.cc:492] T 61923b7b8662455e8b7f8d394dfbf021 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:10.648582 17819 tablet_bootstrap.cc:654] T 61923b7b8662455e8b7f8d394dfbf021 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.639495 17092 tablet_service.cc:1511] Processing CreateTablet for tablet b338603394cb4ee087498b8d63b7ded5 (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 14, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.639148 17219 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 10281ddd7db046fa8f4e275082630dca. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.649740 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet e865b8f545e54b7eb0015f6bc1374540 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.649873 17092 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet b338603394cb4ee087498b8d63b7ded5. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.650120 17813 tablet_replica.cc:333] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:10.651255 17813 raft_consensus.cc:2243] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:10.651238 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.171s user 0.000s sys 0.000s Metrics: {"mutex_wait_us":170493}
I20260929 02:01:10.651218 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.176s user 0.005s sys 0.000s Metrics: {"bytes_written":443,"delete_count":0,"lbm_write_time_us":167,"lbm_writes_lt_1ms":4,"mutex_wait_us":167956,"reinsert_count":0,"update_count":2}
I20260929 02:01:10.651590 17218 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet 93799d6bf3c8407589d9cfb48dd1abd7. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.651582 17813 raft_consensus.cc:2272] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.652410 17813 ts_tablet_manager.cc:1962] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.638635 17224 tablet_service.cc:1511] Processing CreateTablet for tablet ce78e617719b475991ff07a05136c47d (DEFAULT_TABLE table=table-4 [id=6ccec785022f411bb9f66a04facaf321]), partition=HASH (key) PARTITION 13, RANGE (key) PARTITION UNBOUNDED
I20260929 02:01:10.640117 17820 tablet_bootstrap.cc:492] T 1c5034ef0b384105a1d97e01d9fead7d P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:10.658560 17814 raft_consensus.cc:2243] T e865b8f545e54b7eb0015f6bc1374540 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.659068 17814 pending_rounds.cc:70] T e865b8f545e54b7eb0015f6bc1374540 P 8668afa5b5ca4ebdabc812d0789de7bc: Trying to abort 1 pending ops.
I20260929 02:01:10.659358 17814 pending_rounds.cc:77] T e865b8f545e54b7eb0015f6bc1374540 P 8668afa5b5ca4ebdabc812d0789de7bc: Aborting op as it isn't in flight: id { term: 1 index: 1 } timestamp: 7334491217414877184 op_type: NO_OP noop_request { }
I20260929 02:01:10.659897 17814 raft_consensus.cc:2889] T e865b8f545e54b7eb0015f6bc1374540 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: NO_OP replication failed: Aborted: Op aborted
I20260929 02:01:10.660601 17814 raft_consensus.cc:2272] T e865b8f545e54b7eb0015f6bc1374540 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.663929 17814 ts_tablet_manager.cc:1962] T e865b8f545e54b7eb0015f6bc1374540 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.654958 17224 data_dirs.cc:400] Could only allocate 1 dirs of requested 3 for tablet ce78e617719b475991ff07a05136c47d. 1 dirs total, 0 dirs full, 0 dirs failed
I20260929 02:01:10.655673 17820 ts_tablet_manager.cc:1449] T 1c5034ef0b384105a1d97e01d9fead7d P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.078s user 0.004s sys 0.000s
I20260929 02:01:10.665482 17820 raft_consensus.cc:359] T 1c5034ef0b384105a1d97e01d9fead7d P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.665760 17820 raft_consensus.cc:385] T 1c5034ef0b384105a1d97e01d9fead7d P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:10.665885 17820 raft_consensus.cc:740] T 1c5034ef0b384105a1d97e01d9fead7d P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:10.666185 17820 consensus_queue.cc:260] T 1c5034ef0b384105a1d97e01d9fead7d P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.667804 17818 tablet_bootstrap.cc:492] T 61923b7b8662455e8b7f8d394dfbf021 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:10.669020 17820 ts_tablet_manager.cc:1480] T 1c5034ef0b384105a1d97e01d9fead7d P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 02:01:10.670939 17820 tablet_bootstrap.cc:492] T 5663bfc5d2d34aa4b097e6d05e176d16 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:10.671634 17814 ts_tablet_manager.cc:1975] T e865b8f545e54b7eb0015f6bc1374540 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.671833 17814 log.cc:1199] T e865b8f545e54b7eb0015f6bc1374540 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/e865b8f545e54b7eb0015f6bc1374540
I20260929 02:01:10.672340 17814 ts_tablet_manager.cc:1996] T e865b8f545e54b7eb0015f6bc1374540 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:10.672605 17818 tablet_bootstrap.cc:654] T 61923b7b8662455e8b7f8d394dfbf021 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.672739 17820 tablet_bootstrap.cc:654] T 5663bfc5d2d34aa4b097e6d05e176d16 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.673625 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet e865b8f545e54b7eb0015f6bc1374540 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.674422 17814 tablet_replica.cc:333] T 26ea597431574ea9a145a78687442f50 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:10.674647 17814 raft_consensus.cc:2243] T 26ea597431574ea9a145a78687442f50 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.674875 17814 raft_consensus.cc:2272] T 26ea597431574ea9a145a78687442f50 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.675797 17814 ts_tablet_manager.cc:1962] T 26ea597431574ea9a145a78687442f50 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.681905 17813 ts_tablet_manager.cc:1975] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.682066 17813 log.cc:1199] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/e050cc7cbfa445e3be6081ca25c9ed63
I20260929 02:01:10.682480 17813 ts_tablet_manager.cc:1996] T e050cc7cbfa445e3be6081ca25c9ed63 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:10.683113 17814 ts_tablet_manager.cc:1975] T 26ea597431574ea9a145a78687442f50 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.683310 17814 log.cc:1199] T 26ea597431574ea9a145a78687442f50 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/26ea597431574ea9a145a78687442f50
I20260929 02:01:10.683550 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet e050cc7cbfa445e3be6081ca25c9ed63 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.684109 17814 ts_tablet_manager.cc:1996] T 26ea597431574ea9a145a78687442f50 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:10.685611 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 26ea597431574ea9a145a78687442f50 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.686041 17814 tablet_replica.cc:333] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:10.686344 17814 raft_consensus.cc:2243] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:10.687199 17814 raft_consensus.cc:2272] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.688242 17814 ts_tablet_manager.cc:1962] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.689499 17820 tablet_bootstrap.cc:492] T 5663bfc5d2d34aa4b097e6d05e176d16 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:10.689661 17820 ts_tablet_manager.cc:1449] T 5663bfc5d2d34aa4b097e6d05e176d16 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.019s user 0.004s sys 0.000s
I20260929 02:01:10.690402 17820 raft_consensus.cc:359] T 5663bfc5d2d34aa4b097e6d05e176d16 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.690599 17820 raft_consensus.cc:385] T 5663bfc5d2d34aa4b097e6d05e176d16 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:10.690682 17820 raft_consensus.cc:740] T 5663bfc5d2d34aa4b097e6d05e176d16 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:10.690981 17820 consensus_queue.cc:260] T 5663bfc5d2d34aa4b097e6d05e176d16 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.691833 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 5018 bytes on disk
I20260929 02:01:10.692169 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 5018 bytes on disk
I20260929 02:01:10.692498 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 5018 bytes on disk
I20260929 02:01:10.692812 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.001s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":98,"lbm_reads_lt_1ms":4}
I20260929 02:01:10.693162 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.693431 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.001s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.694025 17820 ts_tablet_manager.cc:1480] T 5663bfc5d2d34aa4b097e6d05e176d16 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 02:01:10.694751 17819 tablet_bootstrap.cc:492] T 61923b7b8662455e8b7f8d394dfbf021 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:10.694985 17819 ts_tablet_manager.cc:1449] T 61923b7b8662455e8b7f8d394dfbf021 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.048s user 0.000s sys 0.006s
I20260929 02:01:10.695899 17819 raft_consensus.cc:359] T 61923b7b8662455e8b7f8d394dfbf021 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.696156 17819 raft_consensus.cc:385] T 61923b7b8662455e8b7f8d394dfbf021 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:10.696275 17819 raft_consensus.cc:740] T 61923b7b8662455e8b7f8d394dfbf021 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:10.696610 17819 consensus_queue.cc:260] T 61923b7b8662455e8b7f8d394dfbf021 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.698577 17819 ts_tablet_manager.cc:1480] T 61923b7b8662455e8b7f8d394dfbf021 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.003s user 0.000s sys 0.001s
I20260929 02:01:10.698899 17819 tablet_bootstrap.cc:492] T a3e7696ec7984a8594f0b10fd53cbf71 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:10.701108 17819 tablet_bootstrap.cc:654] T a3e7696ec7984a8594f0b10fd53cbf71 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.703320 17813 tablet_replica.cc:333] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:10.704515 17813 raft_consensus.cc:2243] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:10.704735 17818 tablet_bootstrap.cc:492] T 61923b7b8662455e8b7f8d394dfbf021 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:10.704996 17818 ts_tablet_manager.cc:1449] T 61923b7b8662455e8b7f8d394dfbf021 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.037s user 0.003s sys 0.011s
I20260929 02:01:10.705132 17813 raft_consensus.cc:2272] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.705761 17813 ts_tablet_manager.cc:1962] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.706167 17818 raft_consensus.cc:359] T 61923b7b8662455e8b7f8d394dfbf021 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.706427 17818 raft_consensus.cc:385] T 61923b7b8662455e8b7f8d394dfbf021 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:10.706586 17818 raft_consensus.cc:740] T 61923b7b8662455e8b7f8d394dfbf021 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:10.707026 17814 ts_tablet_manager.cc:1975] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.706948 17818 consensus_queue.cc:260] T 61923b7b8662455e8b7f8d394dfbf021 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.707171 17814 log.cc:1199] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/273fd95488284ef88c2272052f0de976
I20260929 02:01:10.707737 17814 ts_tablet_manager.cc:1996] T 273fd95488284ef88c2272052f0de976 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:10.709291 17818 ts_tablet_manager.cc:1480] T 61923b7b8662455e8b7f8d394dfbf021 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 02:01:10.709673 17818 tablet_bootstrap.cc:492] T 8f12f916b4ee494f94c2007983a440d2 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:10.710875 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 273fd95488284ef88c2272052f0de976 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.712481 17820 tablet_bootstrap.cc:492] T 6037cdfdeaaf4ee29081f8f7d849bf1e P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:10.712679 17814 tablet_replica.cc:333] T 800999ad9c144b02b58cf1bf02b6d9e2 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:10.712917 17814 raft_consensus.cc:2243] T 800999ad9c144b02b58cf1bf02b6d9e2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.713186 17814 raft_consensus.cc:2272] T 800999ad9c144b02b58cf1bf02b6d9e2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.714064 17814 ts_tablet_manager.cc:1962] T 800999ad9c144b02b58cf1bf02b6d9e2 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.715022 17820 tablet_bootstrap.cc:654] T 6037cdfdeaaf4ee29081f8f7d849bf1e P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.715240 17818 tablet_bootstrap.cc:654] T 8f12f916b4ee494f94c2007983a440d2 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.721303 17814 ts_tablet_manager.cc:1975] T 800999ad9c144b02b58cf1bf02b6d9e2 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.721453 17814 log.cc:1199] T 800999ad9c144b02b58cf1bf02b6d9e2 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/800999ad9c144b02b58cf1bf02b6d9e2
I20260929 02:01:10.721987 17814 ts_tablet_manager.cc:1996] T 800999ad9c144b02b58cf1bf02b6d9e2 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:10.722636 17813 ts_tablet_manager.cc:1975] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.722760 17813 log.cc:1199] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/5d8486054d464e4ab45c10da4de28090
I20260929 02:01:10.723138 17813 ts_tablet_manager.cc:1996] T 5d8486054d464e4ab45c10da4de28090 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:10.723362 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 800999ad9c144b02b58cf1bf02b6d9e2 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.723724 17814 tablet_replica.cc:333] T f8dd03b8a32f4ec5917415ee176d272b P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:10.723963 17814 raft_consensus.cc:2243] T f8dd03b8a32f4ec5917415ee176d272b P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.724179 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 5d8486054d464e4ab45c10da4de28090 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.724201 17814 raft_consensus.cc:2272] T f8dd03b8a32f4ec5917415ee176d272b P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.725054 17814 ts_tablet_manager.cc:1962] T f8dd03b8a32f4ec5917415ee176d272b P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.725486 17820 tablet_bootstrap.cc:492] T 6037cdfdeaaf4ee29081f8f7d849bf1e P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:10.725683 17820 ts_tablet_manager.cc:1449] T 6037cdfdeaaf4ee29081f8f7d849bf1e P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.013s user 0.000s sys 0.004s
I20260929 02:01:10.728044 17820 raft_consensus.cc:359] T 6037cdfdeaaf4ee29081f8f7d849bf1e P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.728745 17820 raft_consensus.cc:385] T 6037cdfdeaaf4ee29081f8f7d849bf1e P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:10.728870 17820 raft_consensus.cc:740] T 6037cdfdeaaf4ee29081f8f7d849bf1e P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:10.728958 17813 tablet_replica.cc:333] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:10.729238 17813 raft_consensus.cc:2243] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:10.730206 17818 tablet_bootstrap.cc:492] T 8f12f916b4ee494f94c2007983a440d2 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:10.730252 17819 tablet_bootstrap.cc:492] T a3e7696ec7984a8594f0b10fd53cbf71 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:10.730445 17819 ts_tablet_manager.cc:1449] T a3e7696ec7984a8594f0b10fd53cbf71 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.032s user 0.008s sys 0.003s
I20260929 02:01:10.730494 17818 ts_tablet_manager.cc:1449] T 8f12f916b4ee494f94c2007983a440d2 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.021s user 0.004s sys 0.002s
I20260929 02:01:10.730952 17820 consensus_queue.cc:260] T 6037cdfdeaaf4ee29081f8f7d849bf1e P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.731386 17819 raft_consensus.cc:359] T a3e7696ec7984a8594f0b10fd53cbf71 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.731673 17819 raft_consensus.cc:385] T a3e7696ec7984a8594f0b10fd53cbf71 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:10.731585 17818 raft_consensus.cc:359] T 8f12f916b4ee494f94c2007983a440d2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.731812 17819 raft_consensus.cc:740] T a3e7696ec7984a8594f0b10fd53cbf71 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:10.731925 17818 raft_consensus.cc:385] T 8f12f916b4ee494f94c2007983a440d2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:10.732053 17818 raft_consensus.cc:740] T 8f12f916b4ee494f94c2007983a440d2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:10.732043 17819 consensus_queue.cc:260] T a3e7696ec7984a8594f0b10fd53cbf71 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.732465 17818 consensus_queue.cc:260] T 8f12f916b4ee494f94c2007983a440d2 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.734212 17819 ts_tablet_manager.cc:1480] T a3e7696ec7984a8594f0b10fd53cbf71 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260929 02:01:10.734527 17819 tablet_bootstrap.cc:492] T 6037cdfdeaaf4ee29081f8f7d849bf1e P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:10.734524 17818 ts_tablet_manager.cc:1480] T 8f12f916b4ee494f94c2007983a440d2 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260929 02:01:10.735158 17818 tablet_bootstrap.cc:492] T 3990f27c54a54031a7f554b9633f56f4 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:10.737272 17813 raft_consensus.cc:2272] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.737955 17813 ts_tablet_manager.cc:1962] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.739616 17819 tablet_bootstrap.cc:654] T 6037cdfdeaaf4ee29081f8f7d849bf1e P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.740145 17820 ts_tablet_manager.cc:1480] T 6037cdfdeaaf4ee29081f8f7d849bf1e P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.014s user 0.000s sys 0.003s
I20260929 02:01:10.741195 17818 tablet_bootstrap.cc:654] T 3990f27c54a54031a7f554b9633f56f4 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.743520 17820 tablet_bootstrap.cc:492] T 3843521ef989409da22fdb563bb50689 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:10.744635 17814 ts_tablet_manager.cc:1975] T f8dd03b8a32f4ec5917415ee176d272b P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.744807 17814 log.cc:1199] T f8dd03b8a32f4ec5917415ee176d272b P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/f8dd03b8a32f4ec5917415ee176d272b
I20260929 02:01:10.745492 17814 ts_tablet_manager.cc:1996] T f8dd03b8a32f4ec5917415ee176d272b P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:10.746603 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet f8dd03b8a32f4ec5917415ee176d272b (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
W20260929 02:01:10.746874 17055 consensus_peers.cc:597] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc -> Peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Couldn't send request to peer d103dd4823044faebe46100a0dfbee88. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 964849b43d28439aa45e8a51084bcae9. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:01:10.747013 17814 tablet_replica.cc:333] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:10.747224 17814 raft_consensus.cc:2243] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 8668afa5b5ca4ebdabc812d0789de7bc [term 2 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.747431 17814 raft_consensus.cc:2272] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 8668afa5b5ca4ebdabc812d0789de7bc [term 2 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.747638 17820 tablet_bootstrap.cc:654] T 3843521ef989409da22fdb563bb50689 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.748382 17814 ts_tablet_manager.cc:1962] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.743611 17781 raft_consensus.cc:493] T 61923b7b8662455e8b7f8d394dfbf021 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:10.752225 17781 raft_consensus.cc:515] T 61923b7b8662455e8b7f8d394dfbf021 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.753067 17781 leader_election.cc:290] T 61923b7b8662455e8b7f8d394dfbf021 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:10.754065 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61923b7b8662455e8b7f8d394dfbf021" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:10.754303 17117 raft_consensus.cc:2468] T 61923b7b8662455e8b7f8d394dfbf021 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:10.754294 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61923b7b8662455e8b7f8d394dfbf021" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
W20260929 02:01:10.754935 16921 leader_election.cc:343] T 61923b7b8662455e8b7f8d394dfbf021 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:10.755831 16919 leader_election.cc:304] T 61923b7b8662455e8b7f8d394dfbf021 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters: d103dd4823044faebe46100a0dfbee88
I20260929 02:01:10.756520 17781 raft_consensus.cc:2804] T 61923b7b8662455e8b7f8d394dfbf021 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:10.756611 17781 raft_consensus.cc:493] T 61923b7b8662455e8b7f8d394dfbf021 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:10.756520 17814 ts_tablet_manager.cc:1975] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20260929 02:01:10.756707 17781 raft_consensus.cc:3060] T 61923b7b8662455e8b7f8d394dfbf021 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:10.756773 17814 log.cc:1199] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/2e8c4b957e16467fa6bd6e9c317b1b1f
I20260929 02:01:10.757351 17814 ts_tablet_manager.cc:1996] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:10.758710 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 2e8c4b957e16467fa6bd6e9c317b1b1f (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.758826 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 5018 bytes on disk
I20260929 02:01:10.759119 17814 tablet_replica.cc:333] T 5e9b68ceb78f45edaaebaf7314364446 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:10.759191 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 5018 bytes on disk
I20260929 02:01:10.759464 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling UndoDeltaBlockGCOp(00000000000000000000000000000000): 5018 bytes on disk
I20260929 02:01:10.759521 17781 raft_consensus.cc:515] T 61923b7b8662455e8b7f8d394dfbf021 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.759961 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.759876 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {"cfile_init":1,"lbm_read_time_us":110,"lbm_reads_lt_1ms":4,"spinlock_wait_cycles":1920}
I20260929 02:01:10.759899 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: UndoDeltaBlockGCOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.760468 17814 raft_consensus.cc:2243] T 5e9b68ceb78f45edaaebaf7314364446 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.760586 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61923b7b8662455e8b7f8d394dfbf021" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:10.760797 17117 raft_consensus.cc:3060] T 61923b7b8662455e8b7f8d394dfbf021 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:10.761073 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "61923b7b8662455e8b7f8d394dfbf021" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:10.761251 17814 raft_consensus.cc:2272] T 5e9b68ceb78f45edaaebaf7314364446 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
W20260929 02:01:10.761662 16921 leader_election.cc:343] T 61923b7b8662455e8b7f8d394dfbf021 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:10.761840 17781 leader_election.cc:290] T 61923b7b8662455e8b7f8d394dfbf021 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Requested vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:10.762140 17814 ts_tablet_manager.cc:1962] T 5e9b68ceb78f45edaaebaf7314364446 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.764081 17117 raft_consensus.cc:2468] T 61923b7b8662455e8b7f8d394dfbf021 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:10.764555 16919 leader_election.cc:304] T 61923b7b8662455e8b7f8d394dfbf021 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters: d103dd4823044faebe46100a0dfbee88
I20260929 02:01:10.764950 17781 raft_consensus.cc:2804] T 61923b7b8662455e8b7f8d394dfbf021 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:10.765107 17781 raft_consensus.cc:697] T 61923b7b8662455e8b7f8d394dfbf021 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Becoming Leader. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Running, Role: LEADER
I20260929 02:01:10.765381 17781 consensus_queue.cc:237] T 61923b7b8662455e8b7f8d394dfbf021 P 97df55a0361841319adfae27c9787b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.768014 17818 tablet_bootstrap.cc:492] T 3990f27c54a54031a7f554b9633f56f4 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:10.768841 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.332s user 0.041s sys 0.017s Metrics: {"bytes_written":683,"cfile_cache_hit":29,"cfile_cache_hit_bytes":27619,"cfile_cache_miss":79,"cfile_cache_miss_bytes":45115,"cfile_init":23,"delete_count":0,"delta_iterators_relevant":30,"dirs.queue_time_us":54384,"dirs.run_cpu_time_us":2501,"dirs.run_wall_time_us":16471,"drs_written":1,"lbm_read_time_us":3714,"lbm_reads_lt_1ms":171,"lbm_write_time_us":1154,"lbm_writes_lt_1ms":32,"mutex_wait_us":298,"num_input_rowsets":3,"peak_mem_usage":172190,"reinsert_count":0,"rows_written":88,"spinlock_wait_cycles":8960,"thread_start_us":49892,"threads_started":4,"update_count":16}
I20260929 02:01:10.770634 17818 ts_tablet_manager.cc:1449] T 3990f27c54a54031a7f554b9633f56f4 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.036s user 0.008s sys 0.002s
I20260929 02:01:10.771988 17814 ts_tablet_manager.cc:1975] T 5e9b68ceb78f45edaaebaf7314364446 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.772154 17814 log.cc:1199] T 5e9b68ceb78f45edaaebaf7314364446 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/5e9b68ceb78f45edaaebaf7314364446
I20260929 02:01:10.772396 17820 tablet_bootstrap.cc:492] T 3843521ef989409da22fdb563bb50689 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:10.772392 17819 tablet_bootstrap.cc:492] T 6037cdfdeaaf4ee29081f8f7d849bf1e P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:10.772612 17819 ts_tablet_manager.cc:1449] T 6037cdfdeaaf4ee29081f8f7d849bf1e P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.038s user 0.002s sys 0.002s
I20260929 02:01:10.772614 17820 ts_tablet_manager.cc:1449] T 3843521ef989409da22fdb563bb50689 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.029s user 0.000s sys 0.004s
I20260929 02:01:10.772802 17814 ts_tablet_manager.cc:1996] T 5e9b68ceb78f45edaaebaf7314364446 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:10.773365 17819 raft_consensus.cc:359] T 6037cdfdeaaf4ee29081f8f7d849bf1e P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.773415 17820 raft_consensus.cc:359] T 3843521ef989409da22fdb563bb50689 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.773597 17820 raft_consensus.cc:385] T 3843521ef989409da22fdb563bb50689 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:10.773597 17819 raft_consensus.cc:385] T 6037cdfdeaaf4ee29081f8f7d849bf1e P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:10.773679 17819 raft_consensus.cc:740] T 6037cdfdeaaf4ee29081f8f7d849bf1e P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:10.773679 17820 raft_consensus.cc:740] T 3843521ef989409da22fdb563bb50689 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:10.773938 17820 consensus_queue.cc:260] T 3843521ef989409da22fdb563bb50689 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.774094 17814 tablet_replica.cc:333] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:10.774243 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 5e9b68ceb78f45edaaebaf7314364446 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.774355 17814 raft_consensus.cc:2243] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:10.774422 17820 ts_tablet_manager.cc:1480] T 3843521ef989409da22fdb563bb50689 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 02:01:10.774551 17819 consensus_queue.cc:260] T 6037cdfdeaaf4ee29081f8f7d849bf1e P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.774771 17820 tablet_bootstrap.cc:492] T 8f12f916b4ee494f94c2007983a440d2 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:10.774791 17814 raft_consensus.cc:2272] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.775135 17819 ts_tablet_manager.cc:1480] T 6037cdfdeaaf4ee29081f8f7d849bf1e P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 02:01:10.772154 17818 raft_consensus.cc:359] T 3990f27c54a54031a7f554b9633f56f4 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.775722 17818 raft_consensus.cc:385] T 3990f27c54a54031a7f554b9633f56f4 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:10.775759 17819 tablet_bootstrap.cc:492] T 942ff8f711b849e1a89a807e869354f1 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:10.775915 17818 raft_consensus.cc:740] T 3990f27c54a54031a7f554b9633f56f4 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:10.779551 17820 tablet_bootstrap.cc:654] T 8f12f916b4ee494f94c2007983a440d2 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.780198 17819 tablet_bootstrap.cc:654] T 942ff8f711b849e1a89a807e869354f1 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.780592 17818 consensus_queue.cc:260] T 3990f27c54a54031a7f554b9633f56f4 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.781436 17818 ts_tablet_manager.cc:1480] T 3990f27c54a54031a7f554b9633f56f4 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.010s user 0.002s sys 0.000s
I20260929 02:01:10.781817 17818 tablet_bootstrap.cc:492] T aed0f67930b64ac78c719794f3a3198e P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:10.784371 17814 ts_tablet_manager.cc:1962] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.785774 17820 tablet_bootstrap.cc:492] T 8f12f916b4ee494f94c2007983a440d2 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:10.752027 17813 ts_tablet_manager.cc:1975] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.785961 17813 log.cc:1199] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/e5aa0d7aa540479c89ae9b8b61d89c4e
I20260929 02:01:10.786123 17820 ts_tablet_manager.cc:1449] T 8f12f916b4ee494f94c2007983a440d2 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.002s
I20260929 02:01:10.786135 16816 catalog_manager.cc:5838] T 61923b7b8662455e8b7f8d394dfbf021 P 97df55a0361841319adfae27c9787b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97df55a0361841319adfae27c9787b2f (127.0.71.65). New cstate: current_term: 1 leader_uuid: "97df55a0361841319adfae27c9787b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:10.787010 17820 raft_consensus.cc:359] T 8f12f916b4ee494f94c2007983a440d2 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.787209 17820 raft_consensus.cc:385] T 8f12f916b4ee494f94c2007983a440d2 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:10.787323 17820 raft_consensus.cc:740] T 8f12f916b4ee494f94c2007983a440d2 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:10.787581 17820 consensus_queue.cc:260] T 8f12f916b4ee494f94c2007983a440d2 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.789239 17820 ts_tablet_manager.cc:1480] T 8f12f916b4ee494f94c2007983a440d2 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.003s user 0.001s sys 0.001s
I20260929 02:01:10.789623 17820 tablet_bootstrap.cc:492] T 942ff8f711b849e1a89a807e869354f1 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:10.791626 17813 ts_tablet_manager.cc:1996] T e5aa0d7aa540479c89ae9b8b61d89c4e P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:10.792783 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet e5aa0d7aa540479c89ae9b8b61d89c4e (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.793169 17813 tablet_replica.cc:333] T fed81261ad7d4afabab4ada3e33aa5e2 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:10.793375 17813 raft_consensus.cc:2243] T fed81261ad7d4afabab4ada3e33aa5e2 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.793439 17820 tablet_bootstrap.cc:654] T 942ff8f711b849e1a89a807e869354f1 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.793591 17813 raft_consensus.cc:2272] T fed81261ad7d4afabab4ada3e33aa5e2 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.794342 17813 ts_tablet_manager.cc:1962] T fed81261ad7d4afabab4ada3e33aa5e2 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.794921 17814 ts_tablet_manager.cc:1975] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.795040 17814 log.cc:1199] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/964849b43d28439aa45e8a51084bcae9
I20260929 02:01:10.795425 17814 ts_tablet_manager.cc:1996] T 964849b43d28439aa45e8a51084bcae9 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:10.796366 17814 tablet_replica.cc:333] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:10.796518 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 964849b43d28439aa45e8a51084bcae9 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.796608 17814 raft_consensus.cc:2243] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:10.796952 17814 raft_consensus.cc:2272] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.797570 17814 ts_tablet_manager.cc:1962] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.798537 17819 tablet_bootstrap.cc:492] T 942ff8f711b849e1a89a807e869354f1 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:10.798666 17819 ts_tablet_manager.cc:1449] T 942ff8f711b849e1a89a807e869354f1 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.023s user 0.000s sys 0.003s
I20260929 02:01:10.799409 17819 raft_consensus.cc:359] T 942ff8f711b849e1a89a807e869354f1 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.799598 17819 raft_consensus.cc:385] T 942ff8f711b849e1a89a807e869354f1 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:10.799687 17819 raft_consensus.cc:740] T 942ff8f711b849e1a89a807e869354f1 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:10.799950 17819 consensus_queue.cc:260] T 942ff8f711b849e1a89a807e869354f1 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.800587 17819 ts_tablet_manager.cc:1480] T 942ff8f711b849e1a89a807e869354f1 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 02:01:10.800868 17819 tablet_bootstrap.cc:492] T 8f12f916b4ee494f94c2007983a440d2 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:10.801502 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.801928 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.802279 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.802610 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.804680 17814 ts_tablet_manager.cc:1975] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.804832 17814 log.cc:1199] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/d57762935f614c4391a5236fd2cb72c4
I20260929 02:01:10.804888 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.805203 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.805286 17814 ts_tablet_manager.cc:1996] T d57762935f614c4391a5236fd2cb72c4 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:10.805506 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.806007 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.806241 17814 tablet_replica.cc:333] T b8b094aa976542c2bf9646de8377fe18 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:10.806326 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.806373 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet d57762935f614c4391a5236fd2cb72c4 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.806422 17814 raft_consensus.cc:2243] T b8b094aa976542c2bf9646de8377fe18 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.806609 17814 raft_consensus.cc:2272] T b8b094aa976542c2bf9646de8377fe18 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.806620 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.807159 17814 ts_tablet_manager.cc:1962] T b8b094aa976542c2bf9646de8377fe18 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.808575 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.808873 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.809486 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:10.809809 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:10.809864 17819 tablet_bootstrap.cc:654] T 8f12f916b4ee494f94c2007983a440d2 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.810204 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.810725 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:10.811086 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:10.811463 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:10.811813 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.812125 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.813889 17814 ts_tablet_manager.cc:1975] T b8b094aa976542c2bf9646de8377fe18 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.814020 17814 log.cc:1199] T b8b094aa976542c2bf9646de8377fe18 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/b8b094aa976542c2bf9646de8377fe18
I20260929 02:01:10.814316 17813 ts_tablet_manager.cc:1975] T fed81261ad7d4afabab4ada3e33aa5e2 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.814422 17814 ts_tablet_manager.cc:1996] T b8b094aa976542c2bf9646de8377fe18 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:10.814437 17813 log.cc:1199] T fed81261ad7d4afabab4ada3e33aa5e2 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/fed81261ad7d4afabab4ada3e33aa5e2
I20260929 02:01:10.814832 17813 ts_tablet_manager.cc:1996] T fed81261ad7d4afabab4ada3e33aa5e2 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:10.815423 17814 tablet_replica.cc:333] T e5aa0d7aa540479c89ae9b8b61d89c4e P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:10.815621 17814 raft_consensus.cc:2243] T e5aa0d7aa540479c89ae9b8b61d89c4e P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.815600 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet b8b094aa976542c2bf9646de8377fe18 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.815797 17814 raft_consensus.cc:2272] T e5aa0d7aa540479c89ae9b8b61d89c4e P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.816490 17814 ts_tablet_manager.cc:1962] T e5aa0d7aa540479c89ae9b8b61d89c4e P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.819702 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.017s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":185,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:10.820998 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.014s user 0.006s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":159,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:10.823314 17814 ts_tablet_manager.cc:1975] T e5aa0d7aa540479c89ae9b8b61d89c4e P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.823457 17814 log.cc:1199] T e5aa0d7aa540479c89ae9b8b61d89c4e P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/e5aa0d7aa540479c89ae9b8b61d89c4e
I20260929 02:01:10.823803 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet fed81261ad7d4afabab4ada3e33aa5e2 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.823900 17814 ts_tablet_manager.cc:1996] T e5aa0d7aa540479c89ae9b8b61d89c4e P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:10.822702 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.011s user 0.007s sys 0.001s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":917,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":59,"dirs.run_cpu_time_us":136,"dirs.run_wall_time_us":780,"lbm_read_time_us":162,"lbm_reads_lt_1ms":8,"lbm_write_time_us":198,"lbm_writes_lt_1ms":4,"mutex_wait_us":2297,"reinsert_count":0,"update_count":17}
I20260929 02:01:10.824672 17813 tablet_replica.cc:333] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:10.824963 17814 tablet_replica.cc:333] T 5d8486054d464e4ab45c10da4de28090 P 8668afa5b5ca4ebdabc812d0789de7bc: stopping tablet replica
I20260929 02:01:10.825081 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet e5aa0d7aa540479c89ae9b8b61d89c4e (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.825160 17814 raft_consensus.cc:2243] T 5d8486054d464e4ab45c10da4de28090 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.825399 17814 raft_consensus.cc:2272] T 5d8486054d464e4ab45c10da4de28090 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.825588 17818 tablet_bootstrap.cc:654] T aed0f67930b64ac78c719794f3a3198e P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.825712 17813 raft_consensus.cc:2243] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Raft consensus shutting down.
I20260929 02:01:10.826083 17814 ts_tablet_manager.cc:1962] T 5d8486054d464e4ab45c10da4de28090 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.826083 17813 raft_consensus.cc:2272] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.825984 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.016s user 0.004s sys 0.003s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":917,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"lbm_read_time_us":180,"lbm_reads_lt_1ms":8,"lbm_write_time_us":190,"lbm_writes_lt_1ms":4,"mutex_wait_us":6318,"reinsert_count":0,"update_count":17}
I20260929 02:01:10.827733 17813 ts_tablet_manager.cc:1962] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.827872 17820 tablet_bootstrap.cc:492] T 942ff8f711b849e1a89a807e869354f1 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:10.828279 17820 ts_tablet_manager.cc:1449] T 942ff8f711b849e1a89a807e869354f1 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.039s user 0.006s sys 0.001s
I20260929 02:01:10.829550 17820 raft_consensus.cc:359] T 942ff8f711b849e1a89a807e869354f1 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.829839 17820 raft_consensus.cc:385] T 942ff8f711b849e1a89a807e869354f1 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:10.829931 17820 raft_consensus.cc:740] T 942ff8f711b849e1a89a807e869354f1 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:10.833566 17819 tablet_bootstrap.cc:492] T 8f12f916b4ee494f94c2007983a440d2 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:10.833565 17814 ts_tablet_manager.cc:1975] T 5d8486054d464e4ab45c10da4de28090 P 8668afa5b5ca4ebdabc812d0789de7bc: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.833704 17814 log.cc:1199] T 5d8486054d464e4ab45c10da4de28090 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-1/wal/wals/5d8486054d464e4ab45c10da4de28090
I20260929 02:01:10.833756 17819 ts_tablet_manager.cc:1449] T 8f12f916b4ee494f94c2007983a440d2 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.033s user 0.002s sys 0.003s
I20260929 02:01:10.834198 17814 ts_tablet_manager.cc:1996] T 5d8486054d464e4ab45c10da4de28090 P 8668afa5b5ca4ebdabc812d0789de7bc: Deleting consensus metadata
I20260929 02:01:10.834595 17819 raft_consensus.cc:359] T 8f12f916b4ee494f94c2007983a440d2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.834806 17819 raft_consensus.cc:385] T 8f12f916b4ee494f94c2007983a440d2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:10.834919 17819 raft_consensus.cc:740] T 8f12f916b4ee494f94c2007983a440d2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:10.835062 17818 tablet_bootstrap.cc:492] T aed0f67930b64ac78c719794f3a3198e P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:10.835289 17818 ts_tablet_manager.cc:1449] T aed0f67930b64ac78c719794f3a3198e P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.054s user 0.004s sys 0.006s
I20260929 02:01:10.835446 16800 catalog_manager.cc:5063] TS 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): tablet 5d8486054d464e4ab45c10da4de28090 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.835444 17819 consensus_queue.cc:260] T 8f12f916b4ee494f94c2007983a440d2 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.835992 17819 ts_tablet_manager.cc:1480] T 8f12f916b4ee494f94c2007983a440d2 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.002s user 0.000s sys 0.002s
I20260929 02:01:10.835561 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: CompactRowSetsOp(00000000000000000000000000000000) complete. Timing: real 0.410s user 0.052s sys 0.022s Metrics: {"bytes_written":683,"cfile_cache_hit":45,"cfile_cache_hit_bytes":28093,"cfile_cache_miss":123,"cfile_cache_miss_bytes":51487,"cfile_init":36,"delete_count":0,"delta_iterators_relevant":52,"dirs.queue_time_us":10142,"dirs.run_cpu_time_us":3034,"dirs.run_wall_time_us":18464,"drs_written":1,"lbm_read_time_us":56946,"lbm_reads_10-100_ms":1,"lbm_reads_lt_1ms":266,"lbm_write_time_us":1237,"lbm_writes_lt_1ms":32,"mutex_wait_us":1625,"num_input_rowsets":4,"peak_mem_usage":131072,"reinsert_count":0,"rows_written":88,"thread_start_us":64076,"threads_started":5,"update_count":16}
I20260929 02:01:10.836310 17819 tablet_bootstrap.cc:492] T 5663bfc5d2d34aa4b097e6d05e176d16 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:10.836344 17818 raft_consensus.cc:359] T aed0f67930b64ac78c719794f3a3198e P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.836745 17818 raft_consensus.cc:385] T aed0f67930b64ac78c719794f3a3198e P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:10.836880 17818 raft_consensus.cc:740] T aed0f67930b64ac78c719794f3a3198e P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:10.836989 17820 consensus_queue.cc:260] T 942ff8f711b849e1a89a807e869354f1 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.838729 17820 ts_tablet_manager.cc:1480] T 942ff8f711b849e1a89a807e869354f1 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.010s user 0.003s sys 0.000s
I20260929 02:01:10.839082 17820 tablet_bootstrap.cc:492] T 61923b7b8662455e8b7f8d394dfbf021 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:10.839743 17680 raft_consensus.cc:493] T 1c5034ef0b384105a1d97e01d9fead7d P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:10.839918 17680 raft_consensus.cc:515] T 1c5034ef0b384105a1d97e01d9fead7d P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.840677 17680 leader_election.cc:290] T 1c5034ef0b384105a1d97e01d9fead7d P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:10.841117 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c5034ef0b384105a1d97e01d9fead7d" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:10.841611 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "1c5034ef0b384105a1d97e01d9fead7d" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:10.842120 17820 tablet_bootstrap.cc:654] T 61923b7b8662455e8b7f8d394dfbf021 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
W20260929 02:01:10.842213 17189 leader_election.cc:343] T 1c5034ef0b384105a1d97e01d9fead7d P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
W20260929 02:01:10.842473 17186 leader_election.cc:343] T 1c5034ef0b384105a1d97e01d9fead7d P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:10.842578 17186 leader_election.cc:304] T 1c5034ef0b384105a1d97e01d9fead7d P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f
I20260929 02:01:10.843039 17680 raft_consensus.cc:2749] T 1c5034ef0b384105a1d97e01d9fead7d P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:10.843719 17819 tablet_bootstrap.cc:654] T 5663bfc5d2d34aa4b097e6d05e176d16 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.844326 17820 tablet_bootstrap.cc:492] T 61923b7b8662455e8b7f8d394dfbf021 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:10.844521 17820 ts_tablet_manager.cc:1449] T 61923b7b8662455e8b7f8d394dfbf021 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.006s user 0.005s sys 0.000s
I20260929 02:01:10.837179 17818 consensus_queue.cc:260] T aed0f67930b64ac78c719794f3a3198e P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.845536 17820 raft_consensus.cc:359] T 61923b7b8662455e8b7f8d394dfbf021 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.845738 17820 raft_consensus.cc:385] T 61923b7b8662455e8b7f8d394dfbf021 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:10.845844 17820 raft_consensus.cc:740] T 61923b7b8662455e8b7f8d394dfbf021 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:10.846092 17820 consensus_queue.cc:260] T 61923b7b8662455e8b7f8d394dfbf021 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.847724 17820 ts_tablet_manager.cc:1480] T 61923b7b8662455e8b7f8d394dfbf021 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.003s user 0.002s sys 0.000s
I20260929 02:01:10.848042 17820 tablet_bootstrap.cc:492] T a3e7696ec7984a8594f0b10fd53cbf71 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:10.851706 17818 ts_tablet_manager.cc:1480] T aed0f67930b64ac78c719794f3a3198e P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.016s user 0.002s sys 0.000s
I20260929 02:01:10.851848 17820 tablet_bootstrap.cc:654] T a3e7696ec7984a8594f0b10fd53cbf71 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.853271 17813 ts_tablet_manager.cc:1975] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.853410 17813 log.cc:1199] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/f8dd03b8a32f4ec5917415ee176d272b
I20260929 02:01:10.853844 17813 ts_tablet_manager.cc:1996] T f8dd03b8a32f4ec5917415ee176d272b P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:10.855010 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet f8dd03b8a32f4ec5917415ee176d272b (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.855620 17818 tablet_bootstrap.cc:492] T a3e7696ec7984a8594f0b10fd53cbf71 P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:10.855769 17819 tablet_bootstrap.cc:492] T 5663bfc5d2d34aa4b097e6d05e176d16 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:10.855911 17819 ts_tablet_manager.cc:1449] T 5663bfc5d2d34aa4b097e6d05e176d16 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.020s user 0.003s sys 0.000s
I20260929 02:01:10.856812 17819 raft_consensus.cc:359] T 5663bfc5d2d34aa4b097e6d05e176d16 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.856998 17819 raft_consensus.cc:385] T 5663bfc5d2d34aa4b097e6d05e176d16 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:10.857092 17819 raft_consensus.cc:740] T 5663bfc5d2d34aa4b097e6d05e176d16 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:10.857371 17819 consensus_queue.cc:260] T 5663bfc5d2d34aa4b097e6d05e176d16 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.859790 17819 ts_tablet_manager.cc:1480] T 5663bfc5d2d34aa4b097e6d05e176d16 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 02:01:10.860844 17820 tablet_bootstrap.cc:492] T a3e7696ec7984a8594f0b10fd53cbf71 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:10.861083 17820 ts_tablet_manager.cc:1449] T a3e7696ec7984a8594f0b10fd53cbf71 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.013s user 0.005s sys 0.005s
I20260929 02:01:10.861701 17819 tablet_bootstrap.cc:492] T 3990f27c54a54031a7f554b9633f56f4 P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:10.861982 17820 raft_consensus.cc:359] T a3e7696ec7984a8594f0b10fd53cbf71 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.862247 17820 raft_consensus.cc:385] T a3e7696ec7984a8594f0b10fd53cbf71 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:10.862336 17820 raft_consensus.cc:740] T a3e7696ec7984a8594f0b10fd53cbf71 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:10.862649 17820 consensus_queue.cc:260] T a3e7696ec7984a8594f0b10fd53cbf71 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.863448 17818 tablet_bootstrap.cc:654] T a3e7696ec7984a8594f0b10fd53cbf71 P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.863942 17819 tablet_bootstrap.cc:654] T 3990f27c54a54031a7f554b9633f56f4 P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.866077 17820 ts_tablet_manager.cc:1480] T a3e7696ec7984a8594f0b10fd53cbf71 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.005s user 0.002s sys 0.000s
I20260929 02:01:10.868130 17820 tablet_bootstrap.cc:492] T 4ff036da7c4b4975a2f790b6a80ce050 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:10.868647 17813 tablet_replica.cc:333] T 800999ad9c144b02b58cf1bf02b6d9e2 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:10.868860 17813 raft_consensus.cc:2243] T 800999ad9c144b02b58cf1bf02b6d9e2 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.869071 17813 raft_consensus.cc:2272] T 800999ad9c144b02b58cf1bf02b6d9e2 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.870627 17820 tablet_bootstrap.cc:654] T 4ff036da7c4b4975a2f790b6a80ce050 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.873281 17813 ts_tablet_manager.cc:1962] T 800999ad9c144b02b58cf1bf02b6d9e2 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.875309 17818 tablet_bootstrap.cc:492] T a3e7696ec7984a8594f0b10fd53cbf71 P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:10.875514 17818 ts_tablet_manager.cc:1449] T a3e7696ec7984a8594f0b10fd53cbf71 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.020s user 0.005s sys 0.004s
I20260929 02:01:10.877161 17818 raft_consensus.cc:359] T a3e7696ec7984a8594f0b10fd53cbf71 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.877447 17818 raft_consensus.cc:385] T a3e7696ec7984a8594f0b10fd53cbf71 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:10.877557 17819 tablet_bootstrap.cc:492] T 3990f27c54a54031a7f554b9633f56f4 P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:10.877559 17818 raft_consensus.cc:740] T a3e7696ec7984a8594f0b10fd53cbf71 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:10.877727 17819 ts_tablet_manager.cc:1449] T 3990f27c54a54031a7f554b9633f56f4 P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.016s user 0.002s sys 0.001s
I20260929 02:01:10.877910 17818 consensus_queue.cc:260] T a3e7696ec7984a8594f0b10fd53cbf71 P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.878468 17819 raft_consensus.cc:359] T 3990f27c54a54031a7f554b9633f56f4 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.878708 17819 raft_consensus.cc:385] T 3990f27c54a54031a7f554b9633f56f4 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:10.878788 17819 raft_consensus.cc:740] T 3990f27c54a54031a7f554b9633f56f4 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:10.878865 17820 tablet_bootstrap.cc:492] T 4ff036da7c4b4975a2f790b6a80ce050 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:10.879038 17820 ts_tablet_manager.cc:1449] T 4ff036da7c4b4975a2f790b6a80ce050 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.011s user 0.003s sys 0.007s
I20260929 02:01:10.879078 17819 consensus_queue.cc:260] T 3990f27c54a54031a7f554b9633f56f4 P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.879794 17820 raft_consensus.cc:359] T 4ff036da7c4b4975a2f790b6a80ce050 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.880074 17820 raft_consensus.cc:385] T 4ff036da7c4b4975a2f790b6a80ce050 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:10.880173 17820 raft_consensus.cc:740] T 4ff036da7c4b4975a2f790b6a80ce050 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:10.880460 17820 consensus_queue.cc:260] T 4ff036da7c4b4975a2f790b6a80ce050 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.881800 17819 ts_tablet_manager.cc:1480] T 3990f27c54a54031a7f554b9633f56f4 P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260929 02:01:10.882927 17820 ts_tablet_manager.cc:1480] T 4ff036da7c4b4975a2f790b6a80ce050 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 02:01:10.883689 17819 tablet_bootstrap.cc:492] T 1c5034ef0b384105a1d97e01d9fead7d P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:10.884894 17820 tablet_bootstrap.cc:492] T e15244567c6a4104a45c3d118eeab335 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:10.884927 17398 raft_consensus.cc:493] T a3e7696ec7984a8594f0b10fd53cbf71 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:10.885094 17398 raft_consensus.cc:515] T a3e7696ec7984a8594f0b10fd53cbf71 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.885846 17398 leader_election.cc:290] T a3e7696ec7984a8594f0b10fd53cbf71 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:10.886626 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3e7696ec7984a8594f0b10fd53cbf71" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:10.886905 17820 tablet_bootstrap.cc:654] T e15244567c6a4104a45c3d118eeab335 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.886880 16984 raft_consensus.cc:2468] T a3e7696ec7984a8594f0b10fd53cbf71 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:10.888145 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3e7696ec7984a8594f0b10fd53cbf71" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:10.888171 17056 leader_election.cc:304] T a3e7696ec7984a8594f0b10fd53cbf71 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:10.888343 17253 raft_consensus.cc:2468] T a3e7696ec7984a8594f0b10fd53cbf71 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:10.889556 17819 tablet_bootstrap.cc:654] T 1c5034ef0b384105a1d97e01d9fead7d P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.894060 17813 ts_tablet_manager.cc:1975] T 800999ad9c144b02b58cf1bf02b6d9e2 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.894168 17813 log.cc:1199] T 800999ad9c144b02b58cf1bf02b6d9e2 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/800999ad9c144b02b58cf1bf02b6d9e2
I20260929 02:01:10.894795 17398 raft_consensus.cc:2804] T a3e7696ec7984a8594f0b10fd53cbf71 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:10.894930 17398 raft_consensus.cc:493] T a3e7696ec7984a8594f0b10fd53cbf71 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:10.895017 17398 raft_consensus.cc:3060] T a3e7696ec7984a8594f0b10fd53cbf71 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:10.895885 17820 tablet_bootstrap.cc:492] T e15244567c6a4104a45c3d118eeab335 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:10.896071 17820 ts_tablet_manager.cc:1449] T e15244567c6a4104a45c3d118eeab335 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.011s user 0.002s sys 0.007s
I20260929 02:01:10.896361 17818 ts_tablet_manager.cc:1480] T a3e7696ec7984a8594f0b10fd53cbf71 P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent starting tablet: real 0.021s user 0.004s sys 0.000s
I20260929 02:01:10.896958 17820 raft_consensus.cc:359] T e15244567c6a4104a45c3d118eeab335 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.897161 17820 raft_consensus.cc:385] T e15244567c6a4104a45c3d118eeab335 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:10.897249 17820 raft_consensus.cc:740] T e15244567c6a4104a45c3d118eeab335 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:10.897442 17818 tablet_bootstrap.cc:492] T 10281ddd7db046fa8f4e275082630dca P 8668afa5b5ca4ebdabc812d0789de7bc: Bootstrap starting.
I20260929 02:01:10.897564 17820 consensus_queue.cc:260] T e15244567c6a4104a45c3d118eeab335 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.898494 17398 raft_consensus.cc:515] T a3e7696ec7984a8594f0b10fd53cbf71 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.900518 17820 ts_tablet_manager.cc:1480] T e15244567c6a4104a45c3d118eeab335 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.004s user 0.002s sys 0.000s
I20260929 02:01:10.901808 17398 leader_election.cc:290] T a3e7696ec7984a8594f0b10fd53cbf71 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:10.902302 17820 tablet_bootstrap.cc:492] T aed0f67930b64ac78c719794f3a3198e P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:10.902415 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3e7696ec7984a8594f0b10fd53cbf71" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:10.902657 17253 raft_consensus.cc:3060] T a3e7696ec7984a8594f0b10fd53cbf71 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:10.904618 17820 tablet_bootstrap.cc:654] T aed0f67930b64ac78c719794f3a3198e P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.904806 17818 tablet_bootstrap.cc:654] T 10281ddd7db046fa8f4e275082630dca P 8668afa5b5ca4ebdabc812d0789de7bc: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.905828 17253 raft_consensus.cc:2468] T a3e7696ec7984a8594f0b10fd53cbf71 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:10.906409 17055 leader_election.cc:304] T a3e7696ec7984a8594f0b10fd53cbf71 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:10.906919 17398 raft_consensus.cc:2804] T a3e7696ec7984a8594f0b10fd53cbf71 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:10.907428 17398 raft_consensus.cc:697] T a3e7696ec7984a8594f0b10fd53cbf71 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 LEADER]: Becoming Leader. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Running, Role: LEADER
I20260929 02:01:10.907876 17398 consensus_queue.cc:237] T a3e7696ec7984a8594f0b10fd53cbf71 P 8668afa5b5ca4ebdabc812d0789de7bc [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.910015 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "a3e7696ec7984a8594f0b10fd53cbf71" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:10.910195 17813 ts_tablet_manager.cc:1996] T 800999ad9c144b02b58cf1bf02b6d9e2 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:10.910364 16984 raft_consensus.cc:3060] T a3e7696ec7984a8594f0b10fd53cbf71 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:10.911438 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 800999ad9c144b02b58cf1bf02b6d9e2 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.911804 17813 tablet_replica.cc:333] T 964849b43d28439aa45e8a51084bcae9 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:10.912225 17813 raft_consensus.cc:2243] T 964849b43d28439aa45e8a51084bcae9 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.912415 17813 raft_consensus.cc:2272] T 964849b43d28439aa45e8a51084bcae9 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.913164 17813 ts_tablet_manager.cc:1962] T 964849b43d28439aa45e8a51084bcae9 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.913934 17820 tablet_bootstrap.cc:492] T aed0f67930b64ac78c719794f3a3198e P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:10.914096 17820 ts_tablet_manager.cc:1449] T aed0f67930b64ac78c719794f3a3198e P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.012s user 0.004s sys 0.006s
I20260929 02:01:10.914774 16984 raft_consensus.cc:2468] T a3e7696ec7984a8594f0b10fd53cbf71 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:10.914940 17820 raft_consensus.cc:359] T aed0f67930b64ac78c719794f3a3198e P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.915153 17820 raft_consensus.cc:385] T aed0f67930b64ac78c719794f3a3198e P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:10.915244 17820 raft_consensus.cc:740] T aed0f67930b64ac78c719794f3a3198e P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:10.915894 17820 consensus_queue.cc:260] T aed0f67930b64ac78c719794f3a3198e P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.917791 17819 tablet_bootstrap.cc:492] T 1c5034ef0b384105a1d97e01d9fead7d P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:10.917917 17819 ts_tablet_manager.cc:1449] T 1c5034ef0b384105a1d97e01d9fead7d P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.034s user 0.000s sys 0.005s
I20260929 02:01:10.918870 17820 ts_tablet_manager.cc:1480] T aed0f67930b64ac78c719794f3a3198e P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260929 02:01:10.918785 17819 raft_consensus.cc:359] T 1c5034ef0b384105a1d97e01d9fead7d P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.918991 17819 raft_consensus.cc:385] T 1c5034ef0b384105a1d97e01d9fead7d P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:10.919095 17819 raft_consensus.cc:740] T 1c5034ef0b384105a1d97e01d9fead7d P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:10.919333 17819 consensus_queue.cc:260] T 1c5034ef0b384105a1d97e01d9fead7d P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.920881 17820 tablet_bootstrap.cc:492] T 3990f27c54a54031a7f554b9633f56f4 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:10.921720 17680 raft_consensus.cc:493] T 942ff8f711b849e1a89a807e869354f1 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:10.921900 17680 raft_consensus.cc:515] T 942ff8f711b849e1a89a807e869354f1 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.922628 17680 leader_election.cc:290] T 942ff8f711b849e1a89a807e869354f1 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:10.922865 16816 catalog_manager.cc:5838] T a3e7696ec7984a8594f0b10fd53cbf71 P 8668afa5b5ca4ebdabc812d0789de7bc reported cstate change: term changed from 0 to 1, leader changed from <none> to 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66). New cstate: current_term: 1 leader_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:10.923408 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "942ff8f711b849e1a89a807e869354f1" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
W20260929 02:01:10.924122 17186 leader_election.cc:343] T 942ff8f711b849e1a89a807e869354f1 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:10.924547 17819 ts_tablet_manager.cc:1480] T 1c5034ef0b384105a1d97e01d9fead7d P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.007s user 0.000s sys 0.002s
I20260929 02:01:10.925769 17820 tablet_bootstrap.cc:654] T 3990f27c54a54031a7f554b9633f56f4 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.928817 17819 tablet_bootstrap.cc:492] T aed0f67930b64ac78c719794f3a3198e P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:10.930506 17819 tablet_bootstrap.cc:654] T aed0f67930b64ac78c719794f3a3198e P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.930886 17781 raft_consensus.cc:493] T 8f12f916b4ee494f94c2007983a440d2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:10.931074 17781 raft_consensus.cc:515] T 8f12f916b4ee494f94c2007983a440d2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.931921 17781 leader_election.cc:290] T 8f12f916b4ee494f94c2007983a440d2 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:10.932739 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f12f916b4ee494f94c2007983a440d2" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:10.932999 17253 raft_consensus.cc:2468] T 8f12f916b4ee494f94c2007983a440d2 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:10.933553 16921 leader_election.cc:304] T 8f12f916b4ee494f94c2007983a440d2 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:10.934422 17820 tablet_bootstrap.cc:492] T 3990f27c54a54031a7f554b9633f56f4 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:10.934644 17820 ts_tablet_manager.cc:1449] T 3990f27c54a54031a7f554b9633f56f4 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.014s user 0.007s sys 0.003s
I20260929 02:01:10.934906 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f12f916b4ee494f94c2007983a440d2" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:10.935171 17117 raft_consensus.cc:2468] T 8f12f916b4ee494f94c2007983a440d2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:10.936195 17820 raft_consensus.cc:359] T 3990f27c54a54031a7f554b9633f56f4 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.936480 17820 raft_consensus.cc:385] T 3990f27c54a54031a7f554b9633f56f4 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:10.936612 17820 raft_consensus.cc:740] T 3990f27c54a54031a7f554b9633f56f4 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:10.936935 17820 consensus_queue.cc:260] T 3990f27c54a54031a7f554b9633f56f4 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.937752 17624 raft_consensus.cc:2804] T 8f12f916b4ee494f94c2007983a440d2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:10.937856 17624 raft_consensus.cc:493] T 8f12f916b4ee494f94c2007983a440d2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:10.937963 17624 raft_consensus.cc:3060] T 8f12f916b4ee494f94c2007983a440d2 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:10.939157 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.939401 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.939457 17820 ts_tablet_manager.cc:1480] T 3990f27c54a54031a7f554b9633f56f4 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260929 02:01:10.939625 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.940299 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.940768 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.941133 17680 raft_consensus.cc:493] T 4ff036da7c4b4975a2f790b6a80ce050 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:10.941159 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.941268 17680 raft_consensus.cc:515] T 4ff036da7c4b4975a2f790b6a80ce050 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.941421 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.941680 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.941929 17680 leader_election.cc:290] T 4ff036da7c4b4975a2f790b6a80ce050 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:10.941941 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.942190 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.942649 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff036da7c4b4975a2f790b6a80ce050" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:10.943149 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "4ff036da7c4b4975a2f790b6a80ce050" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
W20260929 02:01:10.943219 17186 leader_election.cc:343] T 4ff036da7c4b4975a2f790b6a80ce050 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:10.943673 17820 tablet_bootstrap.cc:492] T b338603394cb4ee087498b8d63b7ded5 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:10.944027 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.944284 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
W20260929 02:01:10.944401 16921 consensus_peers.cc:597] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f -> Peer d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233): Couldn't send request to peer d103dd4823044faebe46100a0dfbee88. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26ea597431574ea9a145a78687442f50. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:01:10.944471 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.944806 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
W20260929 02:01:10.944834 17189 leader_election.cc:343] T 4ff036da7c4b4975a2f790b6a80ce050 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:10.944962 17189 leader_election.cc:304] T 4ff036da7c4b4975a2f790b6a80ce050 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f
I20260929 02:01:10.945330 17680 raft_consensus.cc:2749] T 4ff036da7c4b4975a2f790b6a80ce050 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:10.945832 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:10.946157 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:10.946415 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:10.946503 17820 tablet_bootstrap.cc:654] T b338603394cb4ee087498b8d63b7ded5 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.946758 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.947033 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.945183 17813 ts_tablet_manager.cc:1975] T 964849b43d28439aa45e8a51084bcae9 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.947264 17813 log.cc:1199] T 964849b43d28439aa45e8a51084bcae9 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/964849b43d28439aa45e8a51084bcae9
I20260929 02:01:10.947587 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:10.947734 17813 ts_tablet_manager.cc:1996] T 964849b43d28439aa45e8a51084bcae9 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:10.947906 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:10.948220 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:10.948583 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.948709 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "942ff8f711b849e1a89a807e869354f1" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:10.948818 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.948931 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 964849b43d28439aa45e8a51084bcae9 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.948953 16984 raft_consensus.cc:2468] T 942ff8f711b849e1a89a807e869354f1 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 0.
I20260929 02:01:10.949431 17189 leader_election.cc:304] T 942ff8f711b849e1a89a807e869354f1 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc
I20260929 02:01:10.950018 17680 raft_consensus.cc:2804] T 942ff8f711b849e1a89a807e869354f1 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:10.950112 17680 raft_consensus.cc:493] T 942ff8f711b849e1a89a807e869354f1 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:10.950217 17680 raft_consensus.cc:3060] T 942ff8f711b849e1a89a807e869354f1 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:10.951211 17813 tablet_replica.cc:333] T 273fd95488284ef88c2272052f0de976 P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:10.951397 17813 raft_consensus.cc:2243] T 273fd95488284ef88c2272052f0de976 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus shutting down.
I20260929 02:01:10.951586 17813 raft_consensus.cc:2272] T 273fd95488284ef88c2272052f0de976 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.951800 17624 raft_consensus.cc:515] T 8f12f916b4ee494f94c2007983a440d2 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.952222 17813 ts_tablet_manager.cc:1962] T 273fd95488284ef88c2272052f0de976 P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:10.952522 17624 leader_election.cc:290] T 8f12f916b4ee494f94c2007983a440d2 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Requested vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:10.953089 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f12f916b4ee494f94c2007983a440d2" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:10.953326 17253 raft_consensus.cc:3060] T 8f12f916b4ee494f94c2007983a440d2 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:10.953487 17680 raft_consensus.cc:515] T 942ff8f711b849e1a89a807e869354f1 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.954329 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "8f12f916b4ee494f94c2007983a440d2" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
I20260929 02:01:10.954514 17117 raft_consensus.cc:3060] T 8f12f916b4ee494f94c2007983a440d2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:10.956506 17253 raft_consensus.cc:2468] T 8f12f916b4ee494f94c2007983a440d2 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:10.957314 16921 leader_election.cc:304] T 8f12f916b4ee494f94c2007983a440d2 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters:
I20260929 02:01:10.957506 17117 raft_consensus.cc:2468] T 8f12f916b4ee494f94c2007983a440d2 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:10.957978 17781 raft_consensus.cc:2804] T 8f12f916b4ee494f94c2007983a440d2 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:10.958218 17781 raft_consensus.cc:697] T 8f12f916b4ee494f94c2007983a440d2 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Becoming Leader. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Running, Role: LEADER
I20260929 02:01:10.958597 17781 consensus_queue.cc:237] T 8f12f916b4ee494f94c2007983a440d2 P 97df55a0361841319adfae27c9787b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.958855 17120 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "942ff8f711b849e1a89a807e869354f1" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
W20260929 02:01:10.959393 17186 leader_election.cc:343] T 942ff8f711b849e1a89a807e869354f1 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:10.959949 17680 leader_election.cc:290] T 942ff8f711b849e1a89a807e869354f1 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Requested vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
W20260929 02:01:10.960160 16919 consensus_peers.cc:597] T 26ea597431574ea9a145a78687442f50 P 97df55a0361841319adfae27c9787b2f -> Peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Couldn't send request to peer 8668afa5b5ca4ebdabc812d0789de7bc. Error code: TABLET_NOT_FOUND (6). Status: Not found: Tablet not found: 26ea597431574ea9a145a78687442f50. This is attempt 1: this message will repeat every 5th retry.
I20260929 02:01:10.960809 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "942ff8f711b849e1a89a807e869354f1" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:10.961625 16984 raft_consensus.cc:3060] T 942ff8f711b849e1a89a807e869354f1 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:10.962023 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.020s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":169,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:10.963193 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.016s user 0.006s sys 0.002s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":994,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":1067,"lbm_read_time_us":166,"lbm_reads_lt_1ms":8,"lbm_write_time_us":212,"lbm_writes_lt_1ms":4,"mutex_wait_us":7726,"reinsert_count":0,"update_count":18}
I20260929 02:01:10.965689 17820 tablet_bootstrap.cc:492] T b338603394cb4ee087498b8d63b7ded5 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:10.965878 17820 ts_tablet_manager.cc:1449] T b338603394cb4ee087498b8d63b7ded5 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.022s user 0.000s sys 0.009s
I20260929 02:01:10.966753 17820 raft_consensus.cc:359] T b338603394cb4ee087498b8d63b7ded5 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.966928 17820 raft_consensus.cc:385] T b338603394cb4ee087498b8d63b7ded5 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:10.967037 17820 raft_consensus.cc:740] T b338603394cb4ee087498b8d63b7ded5 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:10.967311 17820 consensus_queue.cc:260] T b338603394cb4ee087498b8d63b7ded5 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.970177 17820 ts_tablet_manager.cc:1480] T b338603394cb4ee087498b8d63b7ded5 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.004s user 0.000s sys 0.002s
I20260929 02:01:10.971098 16804 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.030s user 0.006s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":162,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:10.972204 17820 tablet_bootstrap.cc:492] T 93799d6bf3c8407589d9cfb48dd1abd7 P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:10.975770 17820 tablet_bootstrap.cc:654] T 93799d6bf3c8407589d9cfb48dd1abd7 P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.975548 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.027s user 0.005s sys 0.002s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":994,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":50,"lbm_read_time_us":170,"lbm_reads_lt_1ms":8,"lbm_write_time_us":172,"lbm_writes_lt_1ms":4,"mutex_wait_us":19002,"reinsert_count":0,"update_count":18}
I20260929 02:01:10.975666 16816 catalog_manager.cc:5838] T 8f12f916b4ee494f94c2007983a440d2 P 97df55a0361841319adfae27c9787b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97df55a0361841319adfae27c9787b2f (127.0.71.65). New cstate: current_term: 1 leader_uuid: "97df55a0361841319adfae27c9787b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
I20260929 02:01:10.977267 17398 raft_consensus.cc:493] T 3990f27c54a54031a7f554b9633f56f4 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:10.978009 17820 tablet_bootstrap.cc:492] T 93799d6bf3c8407589d9cfb48dd1abd7 P d103dd4823044faebe46100a0dfbee88: No bootstrap required, opened a new log
I20260929 02:01:10.977902 17398 raft_consensus.cc:515] T 3990f27c54a54031a7f554b9633f56f4 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.978161 17820 ts_tablet_manager.cc:1449] T 93799d6bf3c8407589d9cfb48dd1abd7 P d103dd4823044faebe46100a0dfbee88: Time spent bootstrapping tablet: real 0.006s user 0.003s sys 0.000s
I20260929 02:01:10.978652 17398 leader_election.cc:290] T 3990f27c54a54031a7f554b9633f56f4 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:10.978953 17820 raft_consensus.cc:359] T 93799d6bf3c8407589d9cfb48dd1abd7 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.979147 17820 raft_consensus.cc:385] T 93799d6bf3c8407589d9cfb48dd1abd7 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:10.979233 17820 raft_consensus.cc:740] T 93799d6bf3c8407589d9cfb48dd1abd7 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Initialized, Role: FOLLOWER
I20260929 02:01:10.979449 17820 consensus_queue.cc:260] T 93799d6bf3c8407589d9cfb48dd1abd7 P d103dd4823044faebe46100a0dfbee88 [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.979915 17820 ts_tablet_manager.cc:1480] T 93799d6bf3c8407589d9cfb48dd1abd7 P d103dd4823044faebe46100a0dfbee88: Time spent starting tablet: real 0.002s user 0.000s sys 0.000s
I20260929 02:01:10.979959 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3990f27c54a54031a7f554b9633f56f4" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:10.980216 17820 tablet_bootstrap.cc:492] T 10281ddd7db046fa8f4e275082630dca P d103dd4823044faebe46100a0dfbee88: Bootstrap starting.
I20260929 02:01:10.980188 16986 raft_consensus.cc:2468] T 3990f27c54a54031a7f554b9633f56f4 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:10.980757 17056 leader_election.cc:304] T 3990f27c54a54031a7f554b9633f56f4 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:10.982540 17820 tablet_bootstrap.cc:654] T 10281ddd7db046fa8f4e275082630dca P d103dd4823044faebe46100a0dfbee88: Neither blocks nor log segments found. Creating new log.
I20260929 02:01:10.983520 17398 raft_consensus.cc:2804] T 3990f27c54a54031a7f554b9633f56f4 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:10.983625 17398 raft_consensus.cc:493] T 3990f27c54a54031a7f554b9633f56f4 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:10.983747 17398 raft_consensus.cc:3060] T 3990f27c54a54031a7f554b9633f56f4 P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:10.986351 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.986521 17398 raft_consensus.cc:515] T 3990f27c54a54031a7f554b9633f56f4 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.986639 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.986843 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.987037 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.987648 16986 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3990f27c54a54031a7f554b9633f56f4" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f"
I20260929 02:01:10.987900 16986 raft_consensus.cc:3060] T 3990f27c54a54031a7f554b9633f56f4 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:10.988778 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.989145 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.989380 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.989779 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:10.989737 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3990f27c54a54031a7f554b9633f56f4" candidate_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:10.990008 17253 raft_consensus.cc:2468] T 3990f27c54a54031a7f554b9633f56f4 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 0.
I20260929 02:01:10.990068 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:10.990154 17813 ts_tablet_manager.cc:1975] T 273fd95488284ef88c2272052f0de976 P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 1.1
I20260929 02:01:10.990329 17813 log.cc:1199] T 273fd95488284ef88c2272052f0de976 P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/273fd95488284ef88c2272052f0de976
I20260929 02:01:10.990357 16808 maintenance_manager.cc:419] P 02c2326bb7f24f5c98809afcc5a36d25: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:10.990644 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.990797 17813 ts_tablet_manager.cc:1996] T 273fd95488284ef88c2272052f0de976 P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:10.990906 16805 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.990919 16986 raft_consensus.cc:2468] T 3990f27c54a54031a7f554b9633f56f4 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 8668afa5b5ca4ebdabc812d0789de7bc in term 1.
I20260929 02:01:10.991454 17056 leader_election.cc:304] T 3990f27c54a54031a7f554b9633f56f4 P 8668afa5b5ca4ebdabc812d0789de7bc [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 2 responses out of 3 voters: 2 yes votes; 0 no votes. yes voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f; no voters:
I20260929 02:01:10.991812 17813 tablet_replica.cc:333] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f: stopping tablet replica
I20260929 02:01:10.991791 17769 raft_consensus.cc:2804] T 3990f27c54a54031a7f554b9633f56f4 P 8668afa5b5ca4ebdabc812d0789de7bc [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:10.991909 16984 raft_consensus.cc:2468] T 942ff8f711b849e1a89a807e869354f1 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate d103dd4823044faebe46100a0dfbee88 in term 1.
I20260929 02:01:10.992029 17813 raft_consensus.cc:2243] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f [term 2 LEADER]: Raft consensus shutting down.
I20260929 02:01:10.992612 17189 leader_election.cc:304] T 942ff8f711b849e1a89a807e869354f1 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc
I20260929 02:01:10.992826 16803 catalog_manager.cc:5063] TS 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): tablet 273fd95488284ef88c2272052f0de976 (table table-3 [id=c7b17b57b52c47b0abd04ea8d68d8cc8]) successfully deleted
I20260929 02:01:10.993384 17680 raft_consensus.cc:2804] T 942ff8f711b849e1a89a807e869354f1 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:10.993577 17680 raft_consensus.cc:697] T 942ff8f711b849e1a89a807e869354f1 P d103dd4823044faebe46100a0dfbee88 [term 1 LEADER]: Becoming Leader. State: Replica: d103dd4823044faebe46100a0dfbee88, State: Running, Role: LEADER
I20260929 02:01:10.993909 17680 consensus_queue.cc:237] T 942ff8f711b849e1a89a807e869354f1 P d103dd4823044faebe46100a0dfbee88 [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:10.994800 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.995087 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.995339 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.995568 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling FlushDeltaMemStoresOp(00000000000000000000000000000000): perf score=1.000000
I20260929 02:01:10.997795 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.998131 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.998404 16732 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:10.999231 17813 raft_consensus.cc:2272] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f [term 2 FOLLOWER]: Raft consensus is shut down!
I20260929 02:01:10.999310 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:10.999643 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:10.999953 16734 maintenance_manager.cc:419] P 463404b2470c42c9b6bd67f027cf5e2b: Scheduling MinorDeltaCompactionOp(00000000000000000000000000000000): perf score=0.002000
I20260929 02:01:11.000631 17813 ts_tablet_manager.cc:1962] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f: Deleting tablet data with delete state TABLET_DATA_DELETED
I20260929 02:01:11.000633 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:11.000665 17781 raft_consensus.cc:493] T 5663bfc5d2d34aa4b097e6d05e176d16 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:11.000872 16731 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.000s user 0.000s sys 0.000s Metrics: {}
I20260929 02:01:11.000836 17781 raft_consensus.cc:515] T 5663bfc5d2d34aa4b097e6d05e176d16 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:11.001677 17781 leader_election.cc:290] T 5663bfc5d2d34aa4b097e6d05e176d16 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:11.002138 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5663bfc5d2d34aa4b097e6d05e176d16" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
W20260929 02:01:11.002898 16919 leader_election.cc:343] T 5663bfc5d2d34aa4b097e6d05e176d16 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:11.003729 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5663bfc5d2d34aa4b097e6d05e176d16" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:11.003971 17253 raft_consensus.cc:2468] T 5663bfc5d2d34aa4b097e6d05e176d16 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:11.004552 16921 leader_election.cc:304] T 5663bfc5d2d34aa4b097e6d05e176d16 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc
I20260929 02:01:11.005142 17781 raft_consensus.cc:2804] T 5663bfc5d2d34aa4b097e6d05e176d16 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:11.005266 17781 raft_consensus.cc:493] T 5663bfc5d2d34aa4b097e6d05e176d16 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:11.005365 17781 raft_consensus.cc:3060] T 5663bfc5d2d34aa4b097e6d05e176d16 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:11.009610 17781 raft_consensus.cc:515] T 5663bfc5d2d34aa4b097e6d05e176d16 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Starting leader election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:11.010964 16807 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.024s user 0.002s sys 0.003s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":142,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:11.011183 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5663bfc5d2d34aa4b097e6d05e176d16" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc"
W20260929 02:01:11.012002 16919 leader_election.cc:343] T 5663bfc5d2d34aa4b097e6d05e176d16 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:11.011933 16806 maintenance_manager.cc:643] P 02c2326bb7f24f5c98809afcc5a36d25: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.021s user 0.000s sys 0.007s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":1016,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":896,"lbm_read_time_us":155,"lbm_reads_lt_1ms":8,"lbm_write_time_us":145,"lbm_writes_lt_1ms":4,"mutex_wait_us":12977,"reinsert_count":0,"update_count":19}
I20260929 02:01:11.012655 17781 leader_election.cc:290] T 5663bfc5d2d34aa4b097e6d05e176d16 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Requested vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:11.013458 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "5663bfc5d2d34aa4b097e6d05e176d16" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88"
I20260929 02:01:11.013679 17253 raft_consensus.cc:3060] T 5663bfc5d2d34aa4b097e6d05e176d16 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:11.013908 16730 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: FlushDeltaMemStoresOp(00000000000000000000000000000000) complete. Timing: real 0.018s user 0.005s sys 0.000s Metrics: {"bytes_written":436,"delete_count":0,"lbm_write_time_us":177,"lbm_writes_lt_1ms":4,"reinsert_count":0,"update_count":1}
I20260929 02:01:11.014343 17818 tablet_bootstrap.cc:492] T 10281ddd7db046fa8f4e275082630dca P 8668afa5b5ca4ebdabc812d0789de7bc: No bootstrap required, opened a new log
I20260929 02:01:11.014518 17818 ts_tablet_manager.cc:1449] T 10281ddd7db046fa8f4e275082630dca P 8668afa5b5ca4ebdabc812d0789de7bc: Time spent bootstrapping tablet: real 0.117s user 0.008s sys 0.002s
I20260929 02:01:11.015296 17818 raft_consensus.cc:359] T 10281ddd7db046fa8f4e275082630dca P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:11.015498 17818 raft_consensus.cc:385] T 10281ddd7db046fa8f4e275082630dca P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:11.015575 17818 raft_consensus.cc:740] T 10281ddd7db046fa8f4e275082630dca P 8668afa5b5ca4ebdabc812d0789de7bc [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 8668afa5b5ca4ebdabc812d0789de7bc, State: Initialized, Role: FOLLOWER
I20260929 02:01:11.015838 17818 consensus_queue.cc:260] T 10281ddd7db046fa8f4e275082630dca P 8668afa5b5ca4ebdabc812d0789de7bc [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:11.016647 17253 raft_consensus.cc:2468] T 5663bfc5d2d34aa4b097e6d05e176d16 P d103dd4823044faebe46100a0dfbee88 [term 1 FOLLOWER]: Leader election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 1.
I20260929 02:01:11.017379 17819 tablet_bootstrap.cc:492] T aed0f67930b64ac78c719794f3a3198e P 97df55a0361841319adfae27c9787b2f: No bootstrap required, opened a new log
I20260929 02:01:11.017557 17819 ts_tablet_manager.cc:1449] T aed0f67930b64ac78c719794f3a3198e P 97df55a0361841319adfae27c9787b2f: Time spent bootstrapping tablet: real 0.089s user 0.008s sys 0.013s
I20260929 02:01:11.018168 16921 leader_election.cc:304] T 5663bfc5d2d34aa4b097e6d05e176d16 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc
I20260929 02:01:11.018342 17819 raft_consensus.cc:359] T aed0f67930b64ac78c719794f3a3198e P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Replica starting. Triggering 0 pending ops. Active config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:11.018437 17624 raft_consensus.cc:493] T 3843521ef989409da22fdb563bb50689 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:11.018553 17819 raft_consensus.cc:385] T aed0f67930b64ac78c719794f3a3198e P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Consensus starting up: Expiring failure detector timer to make a prompt election more likely
I20260929 02:01:11.018616 17624 raft_consensus.cc:515] T 3843521ef989409da22fdb563bb50689 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:11.019438 17624 leader_election.cc:290] T 3843521ef989409da22fdb563bb50689 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649), d103dd4823044faebe46100a0dfbee88 (127.0.71.67:44233)
I20260929 02:01:11.019623 17624 raft_consensus.cc:2804] T 5663bfc5d2d34aa4b097e6d05e176d16 P 97df55a0361841319adfae27c9787b2f [term 1 FOLLOWER]: Leader election won for term 1
I20260929 02:01:11.019793 17624 raft_consensus.cc:697] T 5663bfc5d2d34aa4b097e6d05e176d16 P 97df55a0361841319adfae27c9787b2f [term 1 LEADER]: Becoming Leader. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Running, Role: LEADER
I20260929 02:01:11.020131 17624 consensus_queue.cc:237] T 5663bfc5d2d34aa4b097e6d05e176d16 P 97df55a0361841319adfae27c9787b2f [LEADER]: Queue going to LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 1, Majority size: 2, State: 0, Mode: LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:11.018633 17819 raft_consensus.cc:740] T aed0f67930b64ac78c719794f3a3198e P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Becoming Follower/Learner. State: Replica: 97df55a0361841319adfae27c9787b2f, State: Initialized, Role: FOLLOWER
I20260929 02:01:11.020740 17819 consensus_queue.cc:260] T aed0f67930b64ac78c719794f3a3198e P 97df55a0361841319adfae27c9787b2f [NON_LEADER]: Queue going to NON_LEADER mode. State: All replicated index: 0, Majority replicated index: 0, Committed index: 0, Last appended: 0.0, Last appended by leader: 0, Current term: 0, Majority size: -1, State: 0, Mode: NON_LEADER, active raft config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:11.021286 17819 ts_tablet_manager.cc:1480] T aed0f67930b64ac78c719794f3a3198e P 97df55a0361841319adfae27c9787b2f: Time spent starting tablet: real 0.004s user 0.001s sys 0.001s
I20260929 02:01:11.021468 17682 raft_consensus.cc:493] T b338603394cb4ee087498b8d63b7ded5 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:11.021289 16733 maintenance_manager.cc:643] P 463404b2470c42c9b6bd67f027cf5e2b: MinorDeltaCompactionOp(00000000000000000000000000000000) complete. Timing: real 0.021s user 0.006s sys 0.001s Metrics: {"cfile_cache_miss":4,"cfile_cache_miss_bytes":1016,"cfile_init":1,"delete_count":0,"delta_blocks_compacted":2,"delta_iterators_relevant":2,"dirs.queue_time_us":57,"lbm_read_time_us":159,"lbm_reads_lt_1ms":8,"lbm_write_time_us":159,"lbm_writes_lt_1ms":4,"mutex_wait_us":13332,"reinsert_count":0,"update_count":19}
I20260929 02:01:11.021626 17819 tablet_bootstrap.cc:492] T 10281ddd7db046fa8f4e275082630dca P 97df55a0361841319adfae27c9787b2f: Bootstrap starting.
I20260929 02:01:11.021622 17682 raft_consensus.cc:515] T b338603394cb4ee087498b8d63b7ded5 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
I20260929 02:01:11.022272 17682 leader_election.cc:290] T b338603394cb4ee087498b8d63b7ded5 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:11.023236 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b338603394cb4ee087498b8d63b7ded5" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:11.023864 17819 tablet_bootstrap.cc:654] T 10281ddd7db046fa8f4e275082630dca P 97df55a0361841319adfae27c9787b2f: Neither blocks nor log segments found. Creating new log.
W20260929 02:01:11.023836 17186 leader_election.cc:343] T b338603394cb4ee087498b8d63b7ded5 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:11.024222 17117 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3843521ef989409da22fdb563bb50689" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" is_pre_election: true
I20260929 02:01:11.025295 16984 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "b338603394cb4ee087498b8d63b7ded5" candidate_uuid: "d103dd4823044faebe46100a0dfbee88" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "97df55a0361841319adfae27c9787b2f" is_pre_election: true
I20260929 02:01:11.025477 17682 raft_consensus.cc:493] T 6037cdfdeaaf4ee29081f8f7d849bf1e P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election (no leader contacted us within the election timeout)
I20260929 02:01:11.025619 17682 raft_consensus.cc:515] T 6037cdfdeaaf4ee29081f8f7d849bf1e P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Starting pre-election with config: opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } }
W20260929 02:01:11.026417 16919 leader_election.cc:343] T 3843521ef989409da22fdb563bb50689 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:11.026575 16819 catalog_manager.cc:5838] T 5663bfc5d2d34aa4b097e6d05e176d16 P 97df55a0361841319adfae27c9787b2f reported cstate change: term changed from 0 to 1, leader changed from <none> to 97df55a0361841319adfae27c9787b2f (127.0.71.65). New cstate: current_term: 1 leader_uuid: "97df55a0361841319adfae27c9787b2f" committed_config { opid_index: -1 OBSOLETE_local: false peers { permanent_uuid: "97df55a0361841319adfae27c9787b2f" member_type: VOTER last_known_addr { host: "127.0.71.65" port: 43625 } health_report { overall_health: HEALTHY } } peers { permanent_uuid: "8668afa5b5ca4ebdabc812d0789de7bc" member_type: VOTER last_known_addr { host: "127.0.71.66" port: 42649 } health_report { overall_health: UNKNOWN } } peers { permanent_uuid: "d103dd4823044faebe46100a0dfbee88" member_type: VOTER last_known_addr { host: "127.0.71.67" port: 44233 } health_report { overall_health: UNKNOWN } } }
W20260929 02:01:11.026938 17189 leader_election.cc:343] T b338603394cb4ee087498b8d63b7ded5 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Tablet error from VoteRequest() call to peer 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625): Illegal state: must be running to vote when last-logged opid is not known
I20260929 02:01:11.027073 17189 leader_election.cc:304] T b338603394cb4ee087498b8d63b7ded5 P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate lost. Election summary: received 3 responses out of 3 voters: 1 yes votes; 2 no votes. yes voters: d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc, 97df55a0361841319adfae27c9787b2f
I20260929 02:01:11.028964 17682 leader_election.cc:290] T 6037cdfdeaaf4ee29081f8f7d849bf1e P d103dd4823044faebe46100a0dfbee88 [CANDIDATE]: Term 1 pre-election: Requested pre-vote from peers 97df55a0361841319adfae27c9787b2f (127.0.71.65:43625), 8668afa5b5ca4ebdabc812d0789de7bc (127.0.71.66:42649)
I20260929 02:01:11.029167 17682 raft_consensus.cc:2749] T b338603394cb4ee087498b8d63b7ded5 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election lost for term 1. Reason: could not achieve majority
I20260929 02:01:11.032383 17253 tablet_service.cc:1917] Received RequestConsensusVote() RPC: tablet_id: "3843521ef989409da22fdb563bb50689" candidate_uuid: "97df55a0361841319adfae27c9787b2f" candidate_term: 1 candidate_status { last_received { term: 0 index: 0 } } ignore_live_leader: false dest_uuid: "d103dd4823044faebe46100a0dfbee88" is_pre_election: true
I20260929 02:01:11.032642 17253 raft_consensus.cc:2468] T 3843521ef989409da22fdb563bb50689 P d103dd4823044faebe46100a0dfbee88 [term 0 FOLLOWER]: Leader pre-election vote request: Granting yes vote for candidate 97df55a0361841319adfae27c9787b2f in term 0.
I20260929 02:01:11.033118 16921 leader_election.cc:304] T 3843521ef989409da22fdb563bb50689 P 97df55a0361841319adfae27c9787b2f [CANDIDATE]: Term 1 pre-election: Election decided. Result: candidate won. Election summary: received 3 responses out of 3 voters: 2 yes votes; 1 no votes. yes voters: 97df55a0361841319adfae27c9787b2f, d103dd4823044faebe46100a0dfbee88; no voters: 8668afa5b5ca4ebdabc812d0789de7bc
I20260929 02:01:11.033221 17813 ts_tablet_manager.cc:1975] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f: tablet deleted with delete type TABLET_DATA_DELETED: last-logged OpId 2.1
I20260929 02:01:11.033401 17813 log.cc:1199] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f: Deleting WAL directory at /tmp/dist-test-taskniqJIA/test-tmp/external_mini_cluster-test.0.Kudu3762Test.AddMaster.1790646603355291-285-0/minicluster-data/ts-0/wal/wals/2e8c4b957e16467fa6bd6e9c317b1b1f
I20260929 02:01:11.033577 17624 raft_consensus.cc:2804] T 3843521ef989409da22fdb563bb50689 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Leader pre-election won for term 1
I20260929 02:01:11.033712 17624 raft_consensus.cc:493] T 3843521ef989409da22fdb563bb50689 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Starting leader election (no leader contacted us within the election timeout)
I20260929 02:01:11.033807 17624 raft_consensus.cc:3060] T 3843521ef989409da22fdb563bb50689 P 97df55a0361841319adfae27c9787b2f [term 0 FOLLOWER]: Advancing to term 1
I20260929 02:01:11.033967 17813 ts_tablet_manager.cc:1996] T 2e8c4b957e16467fa6bd6e9c317b1b1f P 97df55a0361841319adfae27c9787b2f: Deleting consensus metadata
I20260929 02:01:11.035315 17813 tablet_replica.cc:333] T 26ea597431574e